Detecting data deduplication opportunities using hash distance

ABSTRACT

Techniques for processing data may include: receiving a candidate data block; computing a distance using a distance function, wherein the distance denotes a measurement of similarity between the candidate data block and a target data block; and determining, using the distance, whether to perform data deduplication of the candidate data block with respect to the target data block to identify at least one sub-block of the candidate data block that is a duplicate of at least one sub-block of the target data block. The distance may be computed using a bit-wise logical exclusive-or operation of the contents of the candidate data block and the target data block. The distance may be computed using a bit-wise logical exclusive-or operation of digests computed for the candidate and target data blocks using a distance preserving hash function. The target and candidate block may be similar if the distance is less than a threshold.

BACKGROUND Technical Field

This application generally relates to data storage and, moreparticularly to data deduplication techniques.

Description of Related Art

Data storage systems are arrangements of hardware and software in whichstorage processors are coupled to arrays of non-volatile storagedevices, such as magnetic disk drives, electronic flash drives, and/oroptical drives. The storage processors service storage requests,arriving from host machines (“hosts”), which specify blocks, files,and/or other data elements to be written, read, created, deleted, and soforth. Software running on the storage processors manages incomingstorage requests and performs various data processing tasks to organizeand secure the data elements on the non-volatile storage devices.

Some storage systems support data “deduplication.” A commondeduplication scheme involves replacing redundant copies of a data blockwith pointers to a single retained copy. Data deduplication may operatein the background, after redundant data blocks have been stored, and/oroperate inline with storage requests. Inline deduplication matches newlyarriving data blocks with previously stored data blocks and configurespointers accordingly, thus avoiding initial storage of redundant copies.

A common deduplication scheme involves computing digests of data blocksand storing the digests in a database. Each digest is computed as a hashof a data block's contents and identifies the data block with a highlevel of uniqueness, even though the digest is typically much smallerthan the data block itself. Digests thus enable block matching toproceed quickly and efficiently, without having to compare blocksdirectly. For each digest, the database stores a pointer that leads to astored version of the respective data block. To perform deduplication ona particular candidate block, a storage system computes a digest of thecandidate block and searches the database for an entry that matches thecomputed digest. If a match is found, the storage system arrangesmetadata of the candidate block to point to the data block that thedatabase has associated with the matching digest. In this manner, aduplicate copy of the data block is avoided.

SUMMARY OF THE INVENTION

Embodiments using the techniques herein may include a method, system,and computer readable medium for processing data comprising: receiving acandidate data block; computing a distance using a distance function,wherein the distance denotes a measurement of similarity between thecandidate data block and a target data block; and determining, using thedistance, whether to perform data deduplication of the candidate datablock with respect to the target data block to identify at least onesub-block of the candidate data block that is a duplicate of at leastone sub-block of the target data block. The distance function maycompute the distance using a logical exclusive-or operation of thecontents of the candidate data block and the target data block. Thedistance function may compute a normalized distance value as thedistance. The processing may include determining whether the distance isless than a threshold distance. The processing may responsive todetermining the distance is less than a threshold distance, determiningthe candidate data block is similar to the target data block andperforming first processing. The first processing may includedetermining whether at least one sub-block of the candidate data blockmatches at least one sub-block of the target data block; and responsiveto determining at least one sub-block of the candidate data blockmatches at least one sub-block of the target data block, performingsecond processing that stores the candidate data block as a partiallydeduplicated data block with at least one sub-block that is a duplicateof another sub-block of the target data block. The processing mayinclude determining whether the distance is zero; and responsive todetermining the distance is zero, determining that the candidate datablock is a duplicate of the target data block and storing the candidatedata block as a duplicate of the target data block. The distancefunction may compute the distance using a bit-wise logical exclusive-oroperation of a first digest and a second digest, wherein the firstdigest may be computed for the candidate data block using a distancepreserving hash function and wherein the second digest may be computedfor the target data block using the distance preserving hash function.The distance function may compute a normalized distance value as thedistance. The processing may include determining whether the distance isless than a threshold distance. The processing may include responsive todetermining the distance is less than a threshold distance, determiningthe candidate data block is similar to the target data block andperforming first processing. The first processing may includedetermining whether at least one sub-block of the candidate data blockmatches at least one sub-block of the target data block; and responsiveto determining at least one sub-block of the candidate data blockmatches at least one sub-block of the target data block, performingsecond processing that stores the candidate data block as a partiallydeduplicated data block with at least one sub-block that is a duplicateof another sub-block of the target data block. The processing mayinclude determining whether a third digest matches a fourth digest,wherein the third digest is computed for the candidate data block usinga second hash function and wherein the fourth digest is computed for thetarget data block using the second hash function, wherein the secondhash function is a stronger hash function than the first hash function;responsive to determining the third digest matches the fourth digest,comparing content of the candidate data block to content of the targetdata block; determining, in accordance with said comparing whether thecandidate data block matches the target data block; and responsive todetermining the candidate data block matches the target data block,storing the candidate data block as a fully deduplicated data block. Thefirst hash function may be a non-cryptographic hash function and thesecond hash function may be a cryptographic hash function. The firsthash function, that may be the distance preserving hash function, may bea cyclic redundancy check (CRC) function that computes a first checksumas the first digest for the candidate data block and computes a secondchecksum as the second digest for the target data block. The method maybe performed as part of inline processing of the candidate data block inconnection with an I/O path or data path when servicing an I/O accessingthe candidate data block. The method may be performed offline and not aspart of inline processing of the candidate data block in connection withan I/O path or data path when servicing an I/O accessing the candidatedata block.

BRIEF DESCRIPTION OF THE DRAWINGS

Features and advantages of the present invention will become moreapparent from the following detailed description of exemplaryembodiments thereof taken in conjunction with the accompanying drawingsin which:

FIG. 1 is a block diagram of an example environment in which embodimentsof the improved technique can be practiced.

FIGS. 2A and 2B are diagrams that show an example sequence foridentifying a duplicate range and a unique range of a candidate blockwhen a candidate sub-block has matched to a target sub-block thatoccupies a first position of a target block.

FIGS. 3A and 3B are diagrams that show an example sequence foridentifying a duplicate range and a unique range of a candidate blockwhen a candidate sub-block has matched to a target sub-block thatoccupies a last position of a target block.

FIGS. 4A, 4B and 4C are diagrams that show an example sequence foridentifying a duplicate range and two discontiguous unique ranges of acandidate block when the duplicate range does not extend all the way toan end of the candidate block.

FIGS. 5A and 5B are diagrams that show an example sequence foridentifying two duplicate ranges and a unique range of a candidate blockwhen two candidate sub-blocks have matched to two different targetblocks.

FIGS. 6A and 6B are diagrams that show an example sequence foridentifying two duplicate ranges and a unique range of a candidate blockwhen two candidate sub-blocks have matched to two distinct ranges of asingle target block.

FIG. 7 is a block diagram of an example arrangement where mappingmetadata is provided in a virtualization layer for supportingdeduplication at sub-block granularity.

FIG. 8 is a block diagram of an example arrangement where mappingmetadata is provided in a mapping layer for supporting deduplication atsub-block granularity.

FIG. 9 is a flowchart that shows an example method of processing I/Owrite requests in the environment of FIG. 1.

FIG. 10 is a flowchart that shows an example method of performing datadeduplication in the environment of FIG. 1.

FIG. 11 is an example illustrating in-line processing as may beperformed in an embodiment in accordance with the techniques herein.

FIGS. 12 and 13 are examples illustrating processing that may beperformed in embodiments in accordance with the techniques herein.

FIGS. 14, 15 and 16 are flowcharts of processing steps that may beperformed in embodiments in accordance with the techniques herein.

DETAILED DESCRIPTION OF EMBODIMENT(S)

Unfortunately, conventional deduplication schemes fail to detectredundant data when blocks are similar but not identical. For example, adata storage system might store multiple copies of a given block, whichare nearly identical except for a timestamp or label, which is unique toeach copy. Owing to the randomizing nature of the hash function, digestscomputed from the slightly different copies of the data block differwidely from one another, such that deduplication is unable to detect theredundant content. Many opportunities to improve storage efficiency aretherefore missed.

In contrast with the prior approach, an improved technique forperforming data deduplication operates at sub-block granularity bysearching a deduplication database for a match between a candidatesub-block of a candidate block and a target sub-block of apreviously-stored target block. When a match is found, the techniqueidentifies a duplicate range shared between the candidate block and thetarget block and effects persistent storage of the duplicate range byconfiguring mapping metadata of the candidate block so that it points tothe duplicate range in the target block.

Advantageously, improved techniques described herein avoid redundantstorage of identical portions of data blocks, even when the data blocksas a whole are different. Storage efficiency is thereby improved.

It should be appreciated that embodiments are provided by way ofnon-limiting examples to illustrate certain features and principles ofthe techniques described herein. However, techniques herein are notlimited to the particular embodiments described.

Described herein are improved techniques for performing datadeduplication that may operate at sub-block granularity by searching adeduplication database for a match between a candidate sub-block of acandidate block and a target sub-block of a previously-stored targetblock.

When a match is found, the technique identifies a duplicate range sharedbetween the candidate block and the target block and effects persistentstorage of the duplicate range by configuring mapping metadata of thecandidate block so that it points to the duplicate range in the targetblock.

FIG. 1 shows an example environment 100 in which embodiments of theimproved techniques herein can be practiced. Here, multiple hosts 110access a data storage system 116 over a network 114. The data storagesystem 116 includes a storage processor, or “SP,” 120 and storage 190,such as magnetic disk drives, electronic flash drives, and/or the like.The data storage system 116 may include multiple SPs (e.g., a second SP120 a). For example, multiple SPs may be provided as circuit boardassemblies or blades, which plug into a chassis that encloses and coolsthe SPs. The chassis has a backplane for interconnecting the SPs, andadditional connections may be made among SPs using cables. In someexamples, the SP 120 is part of a storage cluster, such as one whichcontains any number of storage appliances, where each appliance includesa pair of SPs connected to shared storage devices. In some arrangements,a host application runs directly on the SP (or SPs), such that separatehost machines 110 need not be present. No particular hardwareconfiguration is required, however, as any number of SPs may beprovided, including a single SP, in any arrangement, and the SP 120 canbe any type of computing device capable of running software andprocessing host I/O's.

The network 114 may be any type of network or combination of networks,such as a storage area network (SAN), a local area network (LAN), a widearea network (WAN), the Internet, and/or some other type of network orcombination of networks, for example. In cases where hosts 110 areprovided, such hosts 110 may connect to the SP 120 using varioustechnologies, such as Fibre Channel, iSCSI (Internet small computersystem interface), NFS (network file system), and CIFS (common Internetfile system), for example. As is known, Fibre Channel and iSCSI areblock-based protocols, whereas NFS and CIFS are file-based protocols.The SP 120 is configured to receive I/O requests 112 according toblock-based and/or file-based protocols and to respond to such I/Orequests 112 by reading or writing the storage 190.

The SP 120 includes one or more communication interfaces 122, a set ofprocessing units 124, and memory 130. The communication interfaces 122include, for example, SCSI target adapters and/or network interfaceadapters for converting electronic and/or optical signals received overthe network 114 to electronic form for use by the SP 120. The set ofprocessing units 124 includes one or more processing chips and/orassemblies, such as numerous multi-core CPUs. The memory 130 includesboth volatile memory, e.g., Random Access Memory (RAM), and non-volatilememory, such as one or more ROMs (Read-Only Memories), disk drives,solid state drives, and the like. The set of processing units 124 andthe memory 130 together form control circuitry, which is constructed andarranged to carry out various methods and functions as described herein.Also, the memory 130 includes a variety of software constructs realizedin the form of executable instructions. When the executable instructionsare run by the set of processing units 124, the set of processing units124 is made to carry out the operations of the software constructs.Although certain software constructs are specifically shown anddescribed, it is understood that the memory 130 typically includes manyother software components, which are not shown, such as an operatingsystem, various applications, processes, and daemons.

As further shown in FIG. 1, the memory 130 “includes,” i.e., realizes byexecution of software instructions, a data cache 136, a deduplication(“dedupe”) manager 132, a deduplication database 150, a mapping layer160, a virtualization layer 170, and a physical layer 180. The dedupemanager 132 is configured to manage deduplication activities. The datacache 136 may be realized in volatile memory (e.g., RAM) and isconfigured to store temporarily incoming data arriving from hosts 110.The data cache 136 may also store recently-written and/or read datablocks, to support deduplication. In an example, the data cache 136defines incoming data 138 from hosts 110 in the form of blocks (such asblock 140), where each block is composed of a sequence of sub-blocks(e.g., sub-blocks 140-0 through 140-7). As is known, a “block” is a unitof storage in a data storage system, which generally corresponds to thesmallest unit of storage space that can be allocated. Block sizes varyfrom one storage system to the next, with typical sizes being 4 kB(kilobytes) or 8 kB, for example. In the particular arrangement shown,there are eight sub-blocks per block, and the sub-blocks are contiguousand uniform in size. For example, if the block size is 4 kB, eachsub-block would be 512 B, which corresponds to one sector.

The dedupe database 150 is configured to store digests of sub-blocksalong with associated information. For example, the dedupe database 150stores, for each of multiple sub-blocks, a digest “D” of that sub-blockand a reference (“Ref”) to a location, in the data storage system 116,of a data block that contained the respective sub-block when thedatabase entry was made. The SP 120 computes each digest as a hash ofthe respective sub-block's contents. Thus, different sub-blocks producedifferent digests, with rare hash collisions being possible. Thereference Ref may take various forms. For example, Ref may point to amemory location in the data cache 136 where the data block istemporarily being held. Alternatively, Ref may point to metadata (e.g.,a block virtualization structure) that maps to the data block, such thata persisted version of the data block may be obtained by following thereference to the pointed-to metadata. In some examples, the referencepoints directly to the data block, e.g., in the physical layer 180. Anentry in the dedupe database 150 may further include a sub-blockposition, “Pos,” which indicates a position of the sub-block for thatentry in the referenced data block. For example, if the sub-block for adatabase entry was the first sub-block in the data block that containedit, then the position Pos might be 0. If the sub-block was the lastsub-block in the data block, then the position Pos might be 7. In someexamples, entries in the dedupe database 150 are created only for firstand last sub-blocks of their parent data blocks, such that there areonly two possibilities and Pos may be represented with a single bit. Insome examples, sub-block position may be tracked using a separatebitmap. The bitmap may be arranged by sub-block and may provide a singlebit for each sub-block, to indicate whether the respective sub-block isin the first position or in the last position.

The dedupe database 150 may operate as a memory-resident cache. Thecache may utilize an LRU (least-recently used) eviction policy. In someexamples, when evicting an entry from the dedupe database 150, the SP120 also evicts the data block referenced by that entry in the datacache 136. In some cases, the dedupe database 150 is backed bypersistent storage, with portions read into volatile memory as neededfor fast access.

The mapping layer 160 includes metadata for mapping blocks of dataobjects, such as LUNs (Logical UNits), file systems, virtual machinedisks, and/or the like, which may be managed by a separate namespacelayer (not shown). The mapping layer 160 maps each data object to acorresponding set of block virtualization structures (“BVSs”) in thevirtualization layer 170. In some examples, one BVS is provided for eachaddressable data block in the storage system. The mapping layer 160includes indirect blocks (“Ms”) 162. Each IB 162 is a block thatcontains an array of block pointers, such as 1024 block pointers. In anexample, IBs 162 are arranged in a tree, or in multiple trees, in whichblock pointers in parent Ms point to child Ms and thus multiply thetotal number of BVSs (and therefore data blocks) that can be addressed.

In the virtualization layer 170, the BVSs enable the data storage system116 to deduplicate physical data blocks and to relocate physical datablocks without having to update block pointers in the mapping layer 160.Block pointer updates would be a very burdensome task, given that blockpointers tend to be much more numerous and widely dispersed than areBVSs.

The physical layer 180 manages the physical data blocks of the datastorage system 116. For example, the physical layer 180 is denominatedin data blocks of uniform size, such as 4 kB, 8 kB, or the like. Eachdata block is uniquely addressable.

The physical layer 180 is configured to store host data as storageextents in the data blocks. The extents may hold compressed data oruncompressed data. Some extents may be smaller than a block, e.g., ifthe data are compressed and/or if only a portion of a block's data areuniquely stored. One should appreciate that the physical layer 180 isitself a logical structure and that the bits of data are actually storedin devices of the storage 190. The data storage system 116 may includeadditional layers, which are not shown, such as a RAID (Redundant Arrayof Independent (or Inexpensive) Disks) layer, additional mapping layers,and the like. The depicted arrangement is intended merely to beillustrative.

In example operation, the hosts 110 issue I/O requests 112 to the datastorage system 116. The SP 120 receives the I/O requests 112 at thecommunication interfaces 122 and initiates further processing. Forexample, the SP 120 receives sets of incoming data 138 being written byhosts 110 and renders the data in block-sized increments of memory,referred to herein simply as “blocks.” The data as received from thehosts is not necessarily block-denominated and may arrive in I/O's ofany size. Nevertheless, the data cache 136 may store the arriving dataas blocks, which the data cache 136 may realize in buffer caches, forexample. The size of the buffer caches is preferably configured to matchthe block size of the data storage system 116.

To support sub-block data deduplication, the SP 120 defines multiplesub-blocks for the defined blocks. For example, the SP 120 renders block140 as sub-blocks 140-0 through 140-7, which are contiguous anduniformly sized. The inventors have recognized that a sub-block size of512 B (one sector) is particularly well-suited for sub-blockdeduplication, as many host applications use the sector as a standardsize for performing writes. Although host applications may shift datawhen writing to the data storage system 116, such shifts tend to appearin full-sector increments. Performing deduplication at per-sectorgranularity thus catches many shifted sectors that conventional,block-based deduplication would miss. Setting the sub-block size tosomething different from a sector is not excluded from processingdescribed herein.

With the sub-blocks 140-0 through 140-7 defined, a deduplication attemptbegins by attempting to match one or more the sub-blocks to an entry inthe dedupe database 150. For example, the dedupe manager 132 receives acandidate sub-block 140C of the candidate block 140 and generates adigest of the candidate sub-block 140C, e.g., by applying a hashfunction to the contents of sub-block 140C. The dedupe manager 132 thensearches the dedupe database 150 for a digest D that matches thecomputed digest of sub-block 140C. In a non-limiting example, the dedupedatabase 150 is constructed as a key-value store, where the digest Dforms the key and the reference Ref and position Pos form the value. Insuch cases, searching for a matching entry in the dedupe database 150merely entails looking up the value that corresponds to the digest whichis applied as the key. If no match is found, the dedupe manager 132 maytry a different sub-block of candidate block 140. If no match is foundafter attempting all sub-blocks 140-0 through 140-7, then thededuplication attempt fails for the candidate block 140. The SP 120 maythen allocate a new storage extent from the physical layer 180 and storethe contents of the candidate block 140 in the newly allocated extent.

If the dedupe manager 132 succeeds, however, in finding a matching entry152 to the candidate sub-block 140C, then deduplication proceeds. In anexample, the matching entry 152 is the entry in the dedupe database 150for which the digest DT matches the computed digest of the candidatesub-block 140C. The digest DT itself was computed for an earlierprocessed sub-block, which we refer to as a “target sub-block,” and waspart of an earlier-processed data block, which we refer to as a “targetblock.”

The dedupe manager 132 then follows the reference Ref_(T) in thematching entry to obtain the target block, which may still reside in thedata cache 136 or which may be fetched from storage 190 if it does not.Depending on implementation, the deduplication attempt may fail if thereis a cache miss, as fetching the target block from disk may be tootime-consuming to be warranted.

Assuming the target block is obtained, the dedupe manager 132 mayproceed by confirming the match. For example, the dedupe manager 132compares the candidate sub-block 140C with the target sub-block as readfrom the target block and tests whether the two are the same, e.g., byperforming a data comparison. Note that the matching entry 152indicates, via Pos_(T), the position of the target sub-block in thetarget block. Thus, the comparison may proceed without having to scanthe entire target block.

Assuming the match is confirmed, the dedupe manager 132 may identify thefull extent of the match. For example, a match to the target sub-blockmay be part of a larger match between the candidate block 140 and thetarget block. Given the position, Pos_(T), of the target sub-block inthe target block, the dedupe manager 132 selects a direction in which tosearch for an extended match. For instance, if the target sub-block wasthe first block in the target block, then the search should proceed inthe forward direction. But if the target sub-block was the last block inthe target block, the search should proceed in the reverse direction.The dedupe manager 132 proceeds in the indicated direction, comparingbytes of the candidate block 140 with corresponding bytes of the targetblock, until an end of the candidate block 140 is reached or until thebytes of the two blocks no longer match. The dedupe manager 132 thenidentifies a boundary between a duplicate range of the candidate block,R_(DUP), which is shared with a target range R_(TARG) of the targetblock, and a unique range of the candidate block R_(UQ), which is uniqueto the candidate block 140. Some block comparisons may yield two uniqueranges, one before the duplicate range R_(DUP) and one after.

Next, the dedupe manager 132 effects persistent storage of the candidateblock 140, in a manner that efficiently accounts for both the duplicaterange R_(DUP) and the unique range R_(UQ) (or unique ranges). In theexample shown, the data storage system 116 has already stored the targetblock, along with metadata to support it. For example, SP 120 alreadyconfigured a block pointer 164T in a leaf IB 162T in the mapping layer160. The block pointer 164T is already configured to point to BVS_(T) inthe virtualization layer 170. BVS_(T) in turn points to Extent_(T),which stores the target block in the physical layer 180. Now, to supportstorage of the candidate block 140, the mapping layer 160 configures ablock pointer 164C in leaf IB 162C. The block pointer 164C points toBVSc, which is made to point to two different locations. The firstlocation is the address of Extent_(C), which is designated for storingthe unique range (or ranges), R_(UQ), of the candidate block. The secondlocation is the address of BVS_(T), i.e., the BVS that points to thetarget block and contains the duplicate range, R_(DUP). When pointing toBVS_(T), the metadata in BVS_(C) may specify the range R_(TARG) of thetarget block that contains the shared data, such as by offset andlength. The SP 120 may then store the unique range R_(UQ) in Extent_(C).The stored data may be compressed, if desired.

The described operations thus effect storage of the candidate block 140while consuming only the amount of storage space required to support theunique range, R_(UQ), which may be as small as a single sector, orsmaller if compressed. Some additional metadata may be needed, but theadditional metadata is small compared with the amount of storage spaceconserved. In a conventional, block-based deduplication scheme, theunique range R_(UQ) would have precluded any match to a previous block,so the entire block would have to have been stored, which would havebeen up to eight times the amount of data that is stored in the currentexample.

One significant feature of the described technique is that it combinesdigest-based lookups of sub-blocks with data comparisons of data. Insome circumstances, this arrangement allows data to be deduplicated atlevels even smaller than the size of a sub-block. For example, whencomparing the candidate block 140 with the target block, the dedupemanager 132 may operate with arbitrarily high precision. Thus, thededupe manager 132 can identify boundaries between duplicate ranges andunique ranges with precision as high as a single byte.

Because the described technique uses data comparisons as part of itsprocessing, the act of confirming that there are no hash collisions is asimple matter, as the target block and candidate block are already beingcompared. Some embodiments leverage this feature by using relativelysmall digests in the dedupe database 150, such as digests having fewerthan 128 bits. Although long digests can reduce the risk of hashcollisions to nearly zero, they are complex to compute and can placehigh burdens on processor cores. As the disclosed technique compares theblocks as part of its regular processing, the technique is tolerant tohash collisions. Should a hash collision (false positive match) occur,the collision will be detected by data comparison and treated as anon-match. In such cases, the dedupe manager 132 can simply try again ormove on to the next sub-block of the candidate block, attempting tomatch that one.

Also, one of ordinary skill in the art will appreciate that sub-blockdeduplication is also capable of matching entire blocks. For instance,if a data comparison of a candidate block to a target block results in aduplicate range R_(DUP) the size of the entire block, with a uniquerange R_(UQ) of zero size, then the entire block is matched and nounique range needs to be stored.

One might observe that sub-block duplication generally requires agreater number of digest lookups than does block-based deduplication.However, the use of smaller digests can help to offset the increasednumber of lookups. Also, implementations may limit the cost of lookupswhere no match is found by using Bloom filters, for example.

Although data comparisons are the general rule, it can be avoided insome circumstances. For instance, if the candidate sub-block is in thefirst position of the candidate block (like sub-block 140-0), and theposition indicator Pos_(T) of the matching entry 152 indicates the lastposition in the target block, then the match cannot be extended as thereis no possibility of there being additional shared content. The matchwould thus be limited to the matching sub-block, and a data comparisonis avoided.

The dedupe manager 132 may also play a role in loading the dedupedatabase 150 with new content. For instance, as the data cache 136receives new host data and defines new blocks, the dedupe manager 132may access those blocks and create new database entries for one or moreof their sub-blocks. Creating new entries for all sub-blocks of eachnewly defined block is certainly an option, but we have recognized thatit is generally sufficient to create entries for only the first and thelast sub-blocks, e.g., 140-0 and 140-7. In the example shown, the dedupemanager 132 has created new database entries for sub-blocks 140-0 and140-7. Each entry includes a digest hash of the respective sub-block(D_(C0) or D_(C7)), a position (0 or 7) of the respective sub-block inthe candidate block 140, and a reference Refc to the candidate block140, which may be a pointer to the buffer cache that stores thecandidate block 140 in the data cache 136.

A rationale behind limiting new database entries to first and lastsub-blocks is that data shifts (by sector increments) may be common, butdata inserts may be expected to be relatively rare. Recording entriesfor the first and last sub-blocks ensures that shifted data will produceat least one sub-block match. In some cases, new sub-block entries areavoided for sub-blocks that are already recorded in entries of thededupe database 150.

Referring to FIGS. 2A and 2B, shown is an example sequence fordeduplicating a portion of candidate block 140 when candidate sub-block140C has been matched to a target sub-block 210T in a first position ofa target block 210. As shown in FIG. 2A, the value of Pos_(T) in thematching entry 152 has identified the target sub-block 210T as being inthe first position of the target block 210, which indicates a forwardsearch direction through the candidate block 140. The dedupe manager 132proceeds to search in the forward direction through the candidate block,starting with the candidate sub-block 140C (or the forward end of it,e.g., if a hash collision has been ruled out). As the dedupe manager 132advances through the candidate block 140, it advances correspondinglythrough the target block 210, comparing data of the candidate block 140with data of the target block 210 at a level of granularity smaller thanthe size of a sub-block. For instance, comparisons may proceed in16-byte chunks, in 8-byte chunks, in 1-byte chunks, or in any-sizedchunks are best suited to the implementation.

At some point, the dedupe manager 132 either detects a mismatch orreaches the forward end of the candidate block 140. In this example, nomismatch is detected and, as shown in FIG. 2B, the dedupe manager 132identifies a duplicate range R_(DUP) 220 in the candidate block 140 anda corresponding target range R_(TARG) 230 in the target block 210. Theseranges match byte-for-byte and are exact duplicates of each other. Thededupe manager 132 also identifies a unique range, R_(UQ) 240, of thecandidate block, which does not correspond to any content in the targetblock 210. A boundary 250 may be identified between the duplicate rangeR_(DUP) 220 and the unique range R_(UQ) 240. In this case, the boundary250 aligns with a sub-block boundary, i.e., between sub-blocks 140-1 and140-2. The dedupe manager 132 then proceeds as already described, byarranging mapping metadata to effect storage of the duplicate rangeR_(DUP) (by reference to the range R_(TARG) in the target block) and bydirecting storage of the unique range R_(UQ) in a storage extent in thephysical layer 180. This example thus avoids redundant storage ofR_(DUP) and saves three quarters of a block.

Referring to FIGS. 3A and 3B, shown is an example sequence fordeduplicating a portion of candidate block 140 when candidate sub-block140C has been matched to a target sub-block 310T in a last position of atarget block 310. As shown in FIG. 3A, the value of Pos_(T) in thematching entry 152 has identified the target sub-block 310T as being inthe last position of the target block 310, which indicates a reversesearch direction through the candidate block 140. The dedupe manager 132proceeds to search in the reverse direction through the candidate block,starting with the candidate sub-block 140C (or the back end of it). Asthe dedupe manager 132 retracts through the candidate block 140, itretracts correspondingly through the target block 310, comparing data ofthe candidate block 140 with data of the target block 310 at finegranularity. In this example, no mismatch is detected and, as shown inFIG. 3B, the dedupe manager 132 identifies a duplicate range R_(DUP) 320in the candidate block 140 and a corresponding target range R_(TARG) 330in the target block 310. The dedupe manager 132 also identifies a uniquerange, R_(UQ) 340, of the candidate block 140, which does not correspondto any content in the target block 310. A boundary 350 may also beidentified between the duplicate range R_(DUP) 320 and the unique rangeR_(UQ) 340. In this case, the boundary 350 aligns with a sub-blockboundary, i.e., between sub-blocks 140-2 and 140-3. The process thenproceeds as described above, by arranging mapping metadata to effectstorage of the duplicate range R_(DUP) and by directing storage of theunique range R_(UQ) in the physical layer 180. This example saves threeeighths of a block.

Referring to FIGS. 4A and 4B, shown is a slightly more complex example.As in FIG. 2A, the candidate sub-block 140C has been matched to a targetsub-block 410T in a first position of a target block 410. But here, thecomparison between candidate block 140 and target block 410 has yieldeda mismatch. Rather than extending all the way to the end of thecandidate block 140, as in FIG. 2B, the duplicate range R_(DUP) 420 endsat boundary 452. Beyond this point, the contents of the two blocksdiverge. The boundary 452 does not align with any boundary betweensub-blocks but can rather be found at some byte location relative to thefifth sub-block, 140-5.

As shown in FIG. 4B, three distinct ranges are formed: a duplicate rangeR_(DUP) 420; a first unique range R_(UQ) 440; and a second unique rangeR_(UQ2) 460. Boundary 450 separates R_(DUP) from R_(UQ), and boundary452 separates R_(DUP) from R_(UQ2)·Range R_(DUP) 420 matches targetrange R_(TARG) 430 byte-for-byte.

As shown in FIG. 4C dedupe manager 132 may combine the two uniqueranges, R_(uQ) 440 and R_(uQ2) 460, by placing them adjacently, whilespecifying an insert offset 470 to identify their boundary. The insertoffset range 470 is so named as it identifies the location at which thecontents of R_(DUP) 420 is to be inserted when the candidate block 140is reconstructed, e.g., when a later read request arrives. In anexample, the SP 120 stores the insert offset 470 in mapping metadata forthe candidate block 140. Although not addressed previously, the examplesin FIGS. 2A/2B and 3A/3B may also involve specifying an insert offset,which in those examples would indicate whether the duplicate rangeR_(DUP) is to be inserted before or after the unique range, R_(UQ), whenreconstructing the respective candidate blocks.

Referring to FIGS. 5A and 5B, shown is a slightly more complex examplestill. As shown in FIG. 5A, two candidate sub-blocks, 140C and 140C2, ofthe candidate block 140 have produced respective matches to twodifferent entries in the digest database 150. In this example, thedifferent matches reference different target blocks 510 and 512. Inparticular, candidate sub-block 140C has matched target sub-block 510T,which occupies the last position in target block 510, and candidatesub-block 140C2 has matched target sub-block 512T, which occupies thefirst position of target block 512. As shown in FIG. 5B, the dedupemanager 132 has identified two duplicate ranges, R_(DUP) 520 and RR_(DUP2) 522, with a single unique range R_(UQ) 540 separating the twoduplicate ranges. Duplicate range R_(DUP) 520 matches, byte-for-byte,target range R_(TARG) 530 in target block 510, and duplicate rangeR_(DUP2) 522 matches, byte-for-byte, target range R_(TARG) 532 in targetblock 512. Boundaries 550 and 552 separate the ranges of the candidateblock 140 in the manner shown. As usual, the process stores the uniquerange R_(UQ) 540 in a storage extent in the physical layer 180.

The presence of two duplicate ranges may necessitate the creation ofadditional metadata, as BVSc of the candidate block 140 must point totwo different target ranges 530 and 532 in two different target blocks510 and 512. Two insert offsets may be specified, as well, to indicatewhere each duplicate range is to be inserted relative to the uniquerange.

Referring to FIGS. 6A and 6B, shown is yet another example. This exampleis similar to the one shown in FIGS. 5A and 5B, but here two targetsub-blocks 610T and 610T2 are found within the same target block 610,rather than in two different target blocks. Candidate sub-block 140C hasmatched to target sub-block 610T, and candidate sub-block 140C2 hasmatched to target sub-block 610T2. As shown in FIG. 6B, two duplicateranges are formed, R_(DUP) 620 and R_(DUP2) 622, which match,respectively, with target ranges R_(TARG) 630 and R_(TARG2) 632. Asingle unique range R_(UQ) 640 is formed in the candidate block 140between the two duplicate ranges, with boundaries 650 and 652 identifiedas shown. Unique range R_(UQ) 640 may be stored in the usual way. BVScof the candidate block 140 must now point to two different target ranges630 and 632 in the target block 610, which may require two differentinsert offsets.

FIG. 7 shows the example metadata structures of FIG. 1 in additionaldetail. Here, it is seen that BVSc, which is provided for mapping thecandidate block 140, is located within a BVS block 702. In someexamples, BVS block 702 may include a large number of BVSs. Each BVS hasa reference count, which tracks the number of block pointers that pointto the respective BVS. For instance, BVSc has a reference count of one,because only one block pointer, pointer 164C in leaf IB 162C, points toit. BVST, which is provided for mapping the target block 210, has areference count of two, reflecting the fact that both block pointer 164Tin leaf IB 162T and a pointer in BVSc point to it.

As shown to the right of FIG. 7, BVSc stores an extent structure 710.The extent structure 710 is metadata that references a range of data inthe current block within the data of another block. For example, BVScsupports mapping for a particular logical address of a particular dataobject hosted by the data storage system 116, and BVSc resolves thatlogical address into a first pointer to the unique range R_(UQ) and asecond pointer to the duplicate range R_(DUP). Here, the first pointerof BVSc points to Extent_(C) and the second pointer, which is providedwithin the extent structure 710, points to the target range R_(TARG) ofthe target block 210. Example metadata elements of the extent structure710 include the following:

-   -   Pointer 712. An address of BVST, the BVS of the target block        where the duplicate range resides in the virtualization layer        170.    -   Offset 714. Offset of the target range R_(TARG) in Extent_(T).        Used for locating the starting location of R_(TARG), which        matches R_(DUP).    -   Length 716. The length of the target range R_(TARG) in        Extent_(T). The offset 714 and length 716 together fully        identify the target range R_(TARG) within Extent_(T), which is        itself fully identified by the pointer 712.    -   Insert Offset 718. An offset relative to the unique range R_(UQ)        in Extent_(C) where the contents of the duplicate range R_(DUP)        are to be inserted when reconstituting the candidate block 140.

One should appreciate that an extent structure 710 is required only incases where some of the contents that would normally be contained in anextent pointed to by a BVS have been deduplicated, such that thosecontents reside in a different physical extent from the one pointed toby that BVS. Thus, only some BVSs will use extent structures 710,whereas others do not. In addition, some BVSs may employ multiple extentstructures 710. The examples presented in FIGS. 5A/5B and 6A/6B wouldeach require two extent structures 710, one for identifying each of thetwo duplicate ranges presented in each example. Further, one shouldappreciate that the placement of extent structures 710 in the mappingmetadata may be varied.

FIG. 8 shows another example of an extent structure. Rather than beinglocated in BVSc, as was the case in FIG. 7, the extent structure 810 isinstead located in leaf IB 162C. Placing the extent structure 810 in theleaf IB has the inherent advantage of avoiding multiple hops. In FIG. 7,for example, the SP 120 would have to access BVSc, read its extentstructure 710, and then access BVST. Those accesses would have to beperformed in order, one after the other. By contrast, locating theextent structure 810 in the leaf IB, as shown in FIG. 8, allows the SP120 to access both BVSc and BVS_(T) in parallel, promoting fasterprocessing.

Between zero and two extent structures are needed for each leaf-IE blockpointer, with two extent structures shown here as structures 810 a and810 b. As the leaf IB is itself a block, which has limited space, it maybe preferable in some examples to compress the contents of the leaf IBwhen many extent structures 810 are needed. Each leaf IB 162 includes aheader and an array of block pointers. In an example, the header ismarked with a label that identifies the pointers as compressed oruncompressed, and the array of block pointers (but not the header) iscompressed, along with the associated extent structures 810. In thismanner, ample space may be made available for any required number ofextent structures 810.

FIGS. 9 and 10 show flowcharts of processing steps or methods 900 and1000 that may be carried out in connection with the environment 100. Themethods 900 and 1000 are typically performed, for example, by thesoftware constructs described in connection with FIG. 1, which reside inthe memory 130 of the storage processor 120 and are run by the set ofprocessors 124. The various acts of methods 900 and 1000 may be orderedin any suitable way. Accordingly, embodiments may be constructed inwhich acts are performed in orders different from those illustrated,which may include performing some acts simultaneously.

FIG. 9 shows example method 900 of performing inline deduplication atsub-block granularity. At 910, the SP 120 receives sets of dataspecified in write I/O requests 112 into the data cache 136. The datacache 136 stores the data in block-size buffer caches (blocks). Eachsuch block includes multiple sub-blocks such as sub-blocks 140-0 to140-7.

At 920, the dedupe manager 132 obtains a candidate block 140 from thedata cache 136 and performs various acts relating to deduplication ofthe candidate block 140. For example, at 930, the dedupe manager 132computes digests of at least one, but preferably all, sub-blocks 140-1to 140-7 of the candidate block 140. At 940, the dedupe manager 132searches the dedupe database 150 for each computed digest. At 950, if amatch is found, operation proceeds to 960, whereupon the dedupe manager132 accesses the target block 210 (or any of the other target blocksdescribed above), by following the reference Ref_(T) in the matchingentry 152. The dedupe manager 132 then compares the candidate block 140with the target block 210 and identifies the duplicate range R_(DUP) ofthe candidate block 140, which contains the same data as thecorresponding target range R_(TARG) of the target block 210. The dedupemanager 132 also identifies a unique range R_(UQ) of the candidate block140.

With the duplicate range R_(DUP) and the unique range R_(UQ) identified,operation proceeds to 970, whereupon the SP 120 allocates BVSc for thecandidate block 140, stores R_(UQ) in Extent_(C), and points BVSc toR_(UQ). The SP 120 also effects storage of R_(DUP) by reference, bycreating an extent structure 710 (or 810) that points to range R_(TARG)of the target block 210, e.g., by pointing to BVS_(T) (via pointer 712),specifying the offset 714 and length 716 of R_(TARG) in Extent_(T), andspecifying an insert offset 718 that identifies where R_(DUP) is to beinserted relative to R_(UQ) when reconstructing the candidate block 140.If two distinct duplicate ranges are detected, two extent structures 710(or 810) may be created. In this manner, the data storage system 116accounts for and can later access all the data of the candidate block140 without having to store the duplicate range R_(DUP). At 972, a nextcandidate block 140 is obtained and the operation described in steps920-970 is repeated for the new candidate block. Although candidateblocks may be processed serially, as shown, they may also be processedin parallel, e.g., with multiple threads processing respective candidateblocks simultaneously.

Returning to 950, if no match is found to any of the sub-blocks 140-1 to140-7 of the candidate block 140, operation proceeds to 980, whereuponthe dedupe manager 132 creates new entries in the dedupe database 150for the first and last sub-blocks of the candidate block 140. Creatingnew entries includes establishing pointers Ptr in each new entry to thecandidate block 140 in the data cache 136.

At 990, the SP 120 stores the candidate block 140, without performingdeduplication, by providing a new BVS for the candidate block 140,storing the candidate block 140 in a new storage extent, and pointingthe new BVS to the new storage extent. Data may be stored in compressedor uncompressed form.

Referring to FIG. 10, shown is a method 1000 of performing datadeduplication and summarizes some of the improvements described above.At 1010, a data storage system 116 receives an I/O (Input/Output)request 112 that specifies a write of a set of data 138 to the datastorage system. The data storage system 116 defines a candidate block140 from at least a portion of the set of data 138. The candidate blockincludes multiple uniformly-sized sub-blocks, e.g., 140-0 to 140-7, andthe sub-blocks include a candidate sub-block 140C.

At 1020, a deduplication database 150 is searched for a target sub-blockthat matches the candidate sub-block 140C. For example, the dedupemanager 132 computes a digest of the candidate sub-block 140C andsearches the dedupe database 150 for a matching entry.

At 1030, in response to finding a matching entry 152 in thededuplication database 150 for the target sub-block (e.g., 210T, 310T,etc.), (i) accessing a previously-stored target block (e.g., 210, 310,etc.) that contains the target sub-block, (ii) identifying a sharedrange between the candidate block 140 and the target block for which aduplicate range R_(DUP) of the candidate block matches a target rangeR_(TARG) of the target block, and (iii) effecting persistent storage ofthe duplicate range R_(DUP) of the candidate block 140 by configuringmapping metadata (e.g., 710, 810) of the candidate block 140 toreference the target range R_(TARG) of the target block.

An improved technique has been described for performing datadeduplication, which operates at the sub-block granularity by searchinga deduplication database 150 for a match between a candidate sub-block140C of a candidate block 140 and a target sub-block of apreviously-stored target block. When a match is found, the techniqueidentifies a duplicate range R_(DUP) shared between the candidate blockand the target block and effects persistent storage of the duplicaterange R_(DUP) by configuring mapping metadata of the candidate block 140so that it points to the duplicate range R_(TARG) in the target block.

Although techniques have been described for performing datadeduplication inline with I/O requests 112 (e.g., also described in moredetail in following paragraphs), similar techniques can be applied whenperforming deduplication in the background. Background deduplication canproceed substantially as described above, except that the dedupe manager132 obtains candidate blocks 140 from persistent storage, rather thanfrom incoming I/O requests 112. For example, SP 120 reads a data blockfrom persistent storage, computes digests from sub-blocks of the datablock, and searches the dedupe database 150 for a match to one or moreof the sub-blocks. If the search reveals a match to a previously-storedtarget block, the SP 120 identifies a duplicate range R_(DUP) and aunique range R_(UQ) of the candidate block 140, effecting storage of theduplicate range R_(DUP) by configuring metadata (e.g., an extentstructure 710 or 810 in a BVS or IB) to reference the correspondingrange R_(TARG) in the target block. Also, the SP 120 replaces thestorage extent that holds the previously-stored version of the candidateblock 140 with a new storage extent that stores only the unique rangeR_(UQ) of the candidate block 140. The new storage extent may be writtenin place of the original one, with the residual space being freed, orthe unique range R_(UQ) may be written to a newly allocated storageextent in the physical layer 180, with the entirety of the extentholding the original candidate block being freed.

Also, the above description has indicated that certain structures“reference” or “point to” other structures. One should appreciate thatany such reference or pointing can be direct or indirect, meaning thatintervening pointing structures can also be present. Further, althoughreferenced or pointed-to structures may be identified hereinabove as“blocks,” such blocks need not be persistently-stored versions. Forexample, such blocks may be cached versions of data blocks, ormemory-resident structures that have not yet been stored on persistentstorage devices.

Further, the extent structures 710 or 810 (or more generally the blocklevel metadata) can be adapted to store additional metadata for trackingsmall differences between sub-blocks. For instance, a data comparison ofa candidate block with a target block that extends beyond theinitially-matching candidate and target sub-blocks (such as shown inFIG. 2B) may reveal small differences between adjacent sub-blocks (ofcandidate and target blocks) being compared, which sub-blocks are almostbut not quite identical. Such differences may extend over only a smallnumber of bytes, which could easily fit into available extra space inextent structures or other block level metadata. Thus, additionalimprovements may identify such minor differences (deltas) and store themin metadata along with a pointer to their locations in the block orsub-block. In this manner, the dedupe match can be extended to includethe nearly-matching block or sub-block, which becomes part of theduplicate range R_(DUP) and can thus be deduplicated. During a laterread, the SP 120 can reconstruct the block in the usual way and thenaccess the delta from the metadata and insert the delta in place of theduplicate data at the indicated location, thus restoring the difference.Such an embodiment may be useful where small differences in sub-blocksof different data blocks arise merely from timestamps, labels, or minoredits.

Consistent with discussion herein, the data path or I/O path may becharacterized as the path or flow of I/O data through a system. Forexample, the data or I/O path may be the logical flow through hardwareand software components or layers in connection with a user, such as anapplication executing on a host (e.g., more generally, a data storageclient) issuing I/O commands (e.g., SCSI-based commands, and/orfile-based commands) that read and/or write user data to a data storagesystem, and also receiving a response (possibly including requesteddata) in connection such I/O commands.

The control path, also sometimes referred to as the management path, maybe characterized as the path or flow of data management or controlcommands through a system. For example, the control or management pathmay be the logical flow through hardware and software components orlayers in connection with issuing data storage management command toand/or from a data storage system, and also receiving responses(possibly including requested data) to such control or managementcommands. Such commands may be, for example, to establish or modify dataservices, provision storage, perform user account management, and thelike. For example, commands may be issued over the control path toprovision storage for LUNs, create a snapshot, define conditions of whento create another snapshot, define or establish local and/or remotereplication services, define or modify a schedule for snapshot or otherdata replication services, define a RAID group, obtain data storagemanagement and configuration information for display in a graphical userinterface (GUI) of a data storage management program or application,generally modify one or more aspects of a data storage systemconfiguration, list properties and status information regarding LUNs orother storage objects (e.g., physical and/or logical entities in thedata storage system), and the like.

The data path and control path define two sets of different logical flowpaths. In at least some of the data storage system configurations, atleast part of the hardware and network connections used for each of thedata path and control path may differ. For example, although bothcontrol path and data path may generally use a network forcommunications, some of the hardware and software used may differ. Thedata path and control path each define two separate logical flow paths.

An embodiment of a data storage system in accordance with the techniquesherein may perform different data processing operations or services onstored user data. For example, the data storage system may perform oneor more data reduction operations, such as data deduplication andcompression as discussed herein, as well as other types of operations orservices. Generally, data deduplication and compression techniques areknown in the art and any suitable such technique may be used in anembodiment in accordance with the techniques herein. In at least oneembodiment, the compression technique may be a lossless compressiontechnique such as an algorithm from the Lempel Ziv algorithm family(e.g., LZ77, LZ78, LZW, LZR, and the like). In at least one embodimentin accordance with the techniques herein, data deduplication processingperformed may include digest or hash value computation using analgorithm such as based on the SHA-256 hashing algorithm known in theart. Data deduplication generally refers to removing redundant orduplicate data portions. Data deduplication techniques may includelooking for duplicate sub-blocks whereby only a single instance of thesub-block is retained (stored on physical storage) and where pointers orreferences may be used in connection with duplicate or redundant copies(which reference or identify the single stored instance of the datasub-block).

With reference to FIG. 11, shown is an example 1100 illustratingcomponents that may be included in the data path in at least oneexisting data storage system in accordance with the techniques herein.The example 1100 includes two processor nodes A 1102 a and B 1102 b andassociated software stacks 1104, 1106 of the data path where I/Orequests may be received by either processor node 1102 a or 1102 b. Inthe example 1100, the data path 1104 of the processor node A 1102 aincludes: the frontend (FE) component 1104 a (e.g., an FA or front endadapter) that translates the protocol-specific request into a storagesystem-specific request; a cache layer 1104 b (e.g., data cache) wheredata is temporarily stored; an inline processing layer 1105 a; and abackend (BE) component 1104 c that facilitates movement of the databetween the cache and the permanent non-volatile storage (e.g., back endphysical non-volatile storage devices accessed by the BE components suchas controllers or device interfaces that access the physical storage).During movement of data in and out of the cache layer 1104 b (e.g., suchas in connection with reading and writing data respectively, to thephysical storage 1110 a, 1110 b, inline processing may be performed bythe layer 1105 a. Such inline processing operations of 1105 a may beoptionally performed and may include any one of more data processingoperations in connection with data that is flushed from the cache layer1104 b to the back-end non-volatile physical storage 1110 a, 1110 b, aswell as when retrieving data from the back-end non-volatile physicalstorage 1110 a, 1110 b to be stored in the cache layer 1104 b. In atleast one embodiment, the inline processing may include performingcompression and data duplication. Although in following paragraphsreference may be made to inline processing including compression anddata deduplication, more generally, the inline processing may includeperforming any suitable or desirable data processing operations as partof the I/O or data path (e.g., where such operations may include any ofcompression and data deduplication, as well as any other suitable dataprocessing operation).

In a manner similar to that as described for the data path 1104, thedata path 1106 for the processor node B 1102 b has its own FE component1106 a, cache layer 1106 b, inline processing layer 1105 b, and BEcomponent 1106 c that are respectively similar to the components 1104 a,1104 b, 1105 a and 1104 c. The elements 1110 a, 1110 b denote physicalstorage provisioned for LUNs whereby an I/O may be directed to alocation or logical address to read data from, or write data to, thelogical address. The LUNs 1110 a, 1110 b are examples of storage objectsrepresenting logical storage entities included in an existing datastorage system configuration. Since, in this example, writes directed toLUNs 1110 a, 1110 b may be received for processing by either of thenodes 1102 a and 1102 b, the example 1100 illustrates what may also bereferred to as an active-active configuration.

In connection with a write operation as may be received from a host andprocessed by processor node A 1102 a, the write data may be written tothe cache 1104 b, marked as write pending (WP) denoting it needs to bewritten to physical storage 1110 a, 1110 b and, at a later point intime, the write data may be destaged or flushed from the cache to thephysical storage 1110 a, 1110 b by the BE component 1104 c. The writerequest may be considered complete once the write data has been storedin the cache whereby an acknowledgement regarding the completion may bereturned the host (e.g., by component 1104 a). At various points intime, WP data stored in the cache is flushed or written out to thephysical storage 1110 a, 1110 b. In connection with the inlineprocessing layer 1105 a, prior to storing the original data on thephysical storage 1110 a, 1110 b, compression and data deduplicationprocessing may be performed that converts the original data (as storedin the cache prior to inline processing) to a resulting form (that mayinclude compressed and/or deduplicated portions) which is then writtento the physical storage 1110 a, 1110 b. In at least one embodiment, whendeduplication processing determines that a portion (such as a block orone or more sub-blocks) of the original data is a duplicate of anexisting data already stored on the physical storage 1110 a, 1110 b,that particular portion of the original data is not stored in acompressed form and may rather be stored in its deduplicated form (e.g.,there is no need for compression of a block or sub-blocks determined tobe duplicates of other existing data). If an original data portion isnot a duplicate of an existing portion already stored on the physicalstorage 1110 a, 1110 b, the original data portion may be compressed andstored in its compressed form on the physical storage 1110 a, 1110 b.

In connection with a read operation to read a block of data, adetermination is made as to whether the requested read data block isstored in its original form (in the cache 1104 b or on the physicalstorage 1110 a, 1110 b), or whether the requested read data block waspreviously deduplicated or compressed. If the requested read data block(which is stored in its original decompressed, non-deduplicated form) isin the cache, the read data block is retrieved from the cache 1104 b andreturned to the host. Otherwise, if the requested read data block is notin the cache 1104 b but is stored on the physical storage 1110 a, 1110 bin its original form, the requested data block is read by the BEcomponent 1104 c from the backend storage 1110 a, 1110 b, stored in thecache and then returned to the host.

If any sub-block of the requested read data block was previouslydeduplicated, the original corresponding sub-blocks of the read datablock are recreated and stored in the cache in its original form so thatit can be returned to the host. If the requested read data block, orunique sub-block thereof, was previously compressed, the block (orcompressed portion thereof) is first decompressed prior to sending theread data block to the host. If the compressed read data block orsub-blocks are already stored in the cache, the data may be uncompressedto a temporary or buffer location, the uncompressed data is sent to thehost, and the buffer or temporary location is released. If thecompressed read data block (or compressed sub-blocks thereof) is not inthe cache but stored on the physical storage 1110 a, 1110 b, thecompressed read data block (or compressed portions thereof) may be readfrom the physical storage 1110 a, 1110 b into the cache, uncompressed toa buffer or temporary location, and then returned to the host. Thus,requested read data stored on the physical storage 1110 a, 1110 b may bestored in a deduplicated or compressed form as noted above whereprocessing is performed by the inline processing 1105 a to restore orconvert the deduplicated or compressed form of the data to its originaldata form prior to returning the requested read data to the host.

When the processor performs processing, such as in connection withinline processing 1105 a, 1105 b as noted above, data may be loaded intothe cache. In particular, inline compression (ILC) and inline datadeduplication (ILD) may be performed as part of the inline processing1105 a, 1105 b. In at least one embodiment as noted above, the size of adata block processed by ILC and ILD may be 4K bytes where the block ispartitioned into 8 equal sized portions (e.g., 512 bytes each) referredto as sub-blocks or sectors.

In at least one embodiment in accordance with the techniques herein,following paragraphs may provide examples of deduplication processingperformed inline as part of the I/O or data path, as described above inconnection with ILD processing of FIG. 11. However, it should be notedthat the techniques herein are more generally applicable for use indeduplication performed which is not inline or not part of the I/O ordata path, such when processing a data set offline or in the background.

Described above are techniques that may be used for performing datadeduplication that provide for a smaller or finer level of granularityfor determining matching or duplicate data portions at the sub-block orsector level, sometimes referred to herein as partial deduplication,partial block deduplication, or sub-block deduplication. A data block inembodiments described above include 8 sub-blocks identified assub-blocks 0 through 7. Sub-block 0 for a particular data block may alsobe referred to as the left-most or “L” sub-block and sub-block 7 for aparticular data block may also be referred to as the right most or “R”sub-block. Embodiments of techniques described above for partial orsub-block deduplication, such as in the FIG. 9, include calculating thesub-block or sector hashes for each candidate block and then searchingthe dedupe DB 150 to determine whether a sub-block of the candidatematches an existing L or R sub-block of a target current stored in thededupe DB 150. Such processing may be performed for a candidate blockwhich results in no matches being located in the dedupe DB 150.

What will now be described are additional techniques that may be used tofurther optimize data deduplication processing in an embodiment inaccordance with the techniques herein. The following paragraphs providefor further optimizing deduplication processing, for example, in effortsto reduce computational resources when performing partial or sub-blockdeduplication processing for candidate blocks. In at least oneembodiment in accordance with the techniques herein, a distance functionmay be used to evaluate whether a particular target block is expected tohave any sub-block that will match a sub-block of a candidate block. Thedistance function may be used to provide a measurement or degree ofsimilarity between two data blocks, such as the candidate block and aparticular target block. Thus, in at least one embodiment in accordancewith the techniques herein, a decision may be made using a distancevalue computed using a distance function as to whether to look forpartial or sub-block matches for the candidate block within thatparticular target block. In at least one embodiment, a threshold may bedefined for use with computed distances. If a distance computed withrespect to two data blocks using the distance function is less than thethreshold, the two data blocks may be similar and expected to have oneor more matching sub-blocks. Otherwise, if the distance computed withrespect to two data block using the distance function is not less thanthe threshold, the two data blocks may not be similar and are notexpected to have one or more matching sub-blocks.

In the following paragraphs and examples, reference is made to datablocks that are each 4K bytes in size and wherein each of the datablocks includes 8 sub-blocks or sectors (e.g., each sub-block or sectorbeing 512K bytes) for purposes of illustration. Generally, each datablock may be any suitable size and may include any suitable number ofsub-blocks, where each sub-block may be any suitable size.

Reference is made back to the dedupe DB 150 of FIG. 1. In at least oneembodiment, the dedupe DB 150 may be stored in a portion of the memory130 used as a cached copy of the dedupe DB 150. The dedupe DB 150 storedin the memory, such as a cache, may be only a subset or portion of theactual total number of entries of the on-disk dedupe DB as storedpersistently on non-volatile physical storage. In at least oneembodiment, deduplication processing may be performed using only theportion of the dedupe DB currently in the memory or the cache withoutretrieving other portions of the dedupe DB from the non-volatilestorage. In other words, if a look up for a digest of a sub-block isperformed and there is no matching entry in the cached portion of thededupe DB 150, in such an embodiment there is no further lookup in theon-disk dedupe DB and processing determines that there is no entry inthe dedupe DB for digest (e.g., sub-block is unique and not aduplicate).

In an embodiment in accordance with the techniques herein, the dedupe DB150 may include entries for the L and R sub-blocks of each target blockas described above in connection with FIG. 1. Additionally, the dedupeDB 150 may also include entries for the digest of each complete or fulltarget data block (e.g., formed using all 8 sub-blocks of the targetblock). Thus, each entry of the dedupe DB 150 may include a digest, anassociated digest key type of L (for digests of the L sub-block), R (fordigests of the R sub-block) or F (for digests of the full target datablock), and a reference (ref), to a location in the data storage system,of a data block. More generally, the reference field of an entry that isof one of the types L, R or F maps to a location in some form of memoryor storage containing a data block whereby the reference field is usedto access the actual content stored in the data block. The foregoing andother features are described below in more detail.

If an entry of the dedupe DB has a digest key type of L, the entryincludes a digest of an L sub-block of a particular target data blockstored at the location identified by the reference field of the entry.If an entry of the dedupe DB has a digest key type of R, the entryincludes a digest of an R sub-block of a particular target data blockstored at the location identified by the reference field of the entry.If an entry of the dedupe DB has a digest key type of F, the entryincludes a digest of a full target data block and the content of thetarget data block having the digest may be obtained using the referencefield of the entry.

In at least one embodiment in accordance with the techniques herein,deduplication processing may include first performing full blockdeduplication processing to determine whether there is a full orcomplete data block match between a target block and a candidate blockfor which deduplication processing is performed. Responsive todetermining there is no full or complete match between a target blockand a candidate block, partial or sub-block deduplication processing maybe performed.

The digest entries of the dedupe DB 150 with an associated F digest keytype may denote digests used in connection with performing the fullblock deduplication processing in at least one embodiment in accordancewith the techniques herein. The hash function or algorithm used togenerate the full data block or F type digests in an embodiment maygenerally be any suitable function having an acceptable expected hashcollision probability or level of hash resistance. In at least oneembodiment, the hash function used to compute full block digests of Ftype entries of the dedupe DB, and more generally, compute full blockdigests of data blocks for full block deduplication processing may be acryptographic hash function such as SHA-1 or SHA-256.

When performing full block deduplication processing for a candidateblock, at least one embodiment of the techniques herein may includecomputing the full block digest for the candidate block and comparingthe computed full block digest for the candidate to the F type digestentries of the dedupe DB 150 to determine whether there is a matchbetween the computed full block digest for the candidate and a digest ofan F-type entry of a target block of the dedupe cache 150. If a match isfound, processing may then be performed to compare (e.g., byte-by-byte)the content of the candidate block with the content of the target blockhaving the matching F-type entry of the dedupe cache 150 and ensure theactual data contents of the candidate and target blocks match. If theactual contents of the candidate and target blocks match, processingthen proceeds to store the candidate block as a full deduplicated block(e.g., duplicate of target block). If no complete full block match isfound between a target block and the candidate block, partial orsub-block deduplication may be performed.

Consistent with discussion elsewhere herein, for a candidate block thatis found to be unique (e.g., whereby there is no matching target blockand no matching partial or sub-block deduplication performed), thecandidate block may be stored as a new target block and processing mayinclude persistently storing the content of the new target block andadditionally storing digests for the new target block in the dedupe DB150. In at least one embodiment, such processing for the new targetblock may include creating corresponding L, R and F entries for the newtarget block in the dedupe DB 150.

As noted above, an embodiment in accordance with the techniques hereinmay use a distance function to provide a measurement or degree ofsimilarity between two data blocks, such as a candidate block and aparticular target block. In at least one embodiment, the logicalexclusive-or (XOR) function or operation may be used in connection withdefining a distance function D(x,y) that provides a measurement ofsimilarities with respect to the data content of two data blocks, x andy. More formally, in at least one embodiment, D(x,y) may be a numericalvalue that is a distance measurement with respect to a pair of datablocks, x, and y, where the following properties hold:

1. D (x,y)>0.

2. D (x,y)=0 if, and only if x=y.

3. D (x,y)=D (y,x).

Furthermore for an additional data block z,

4. D (x,y)<D (x,z)+D (z,y)

In connection with the techniques herein where D is determined withrespect to data blocks x and y, z may be any arbitrary data block in thesame data space as the data blocks x and y.

In at least one embodiment, D(x,y) may be defined as a distance functionusing the XOR function or operation applied to respective bits/bytes ofdata of the two data blocks, x and y. The distance function D(x,y) maybe defined as a normalized value based on the number of total number ofdata cells or positions, such as number of sub-blocks or sectors, in thedata blocks x and y. More formally, D(x,y) may be defined as:D(x,y)=(Σ_(i=0) ^(N-1) x(i)XOR y(i))/N  EQUATION 1where

N is the number of sub-blocks or sectors in the data blocks x and y,assuming 0-based sub-block numbering;

x(i) denotes the data content of sub-block or sector “i” in data blockx;

y(i) denotes the data content of sub-block or sector “i” in data blocky; and

XOR denotes the logical exclusive OR operation as applied tocorresponding bit positions of the two corresponding sub-blocks orsectors of data, x(i) and y(i). The following summarizes the logical XORoperation as known in the art based on 2 binary input values:

Input values XOR result 0 0 0 1 0 1 0 1 1 1 1 0

Thus at the sub-block level, a resulting XOR value of 0 for the i^(th)sub-block denotes that contents of the i^(th) sub-block of B1 and T1 arethe same, and a value of 1 for the i^(th) sub-block denotes thatcontents of the i^(th) sub-block of B1 and T1 are different.

It should be noted that the numerator and the denominator of EQUATION 1are with respect to the total number of sub-blocks in the data blocks B1and T1, where the numerator denotes how many of the 8 sub-blocks of B1and T1 are different. More generally, an embodiment may compute thedistance with respect to the total number of bit positions or bytepositions in the data blocks B1 and T1, or any suitable groupings of thebits.

The distance computed using EQUATION 1 may be characterized as anormalized value based on the total number of sub-blocks or sectors ineach data block. It will be appreciated by those skilled in the art thatfurther variations of techniques herein may use computed distances thatare not normalized. For example, an embodiment in accordance with thetechniques herein may simply use the numerator computed based onEQUATION 1 as the computed distance values where such computed distancesare not normalized. In such an embodiment, the specified thresholddiscussed elsewhere herein may be an integer threshold quantity.

In at least one embodiment in accordance with the techniques herein, adecision may be made using the distance function D(x,y), wherex=candidate block B1 and y=target block T1, denoting a measurement ofsimilarity between B1 and T1, as to whether to look for partial orsub-block matches for the candidate block B1 within that particulartarget block T1. For example, lower distance values may denote a higherdegree of similarity between B1 and T1. If D(B1, T1) is less than aspecified threshold distance denoting a threshold level of similarity,then it may be determined that B1 and T1 have at least the specifiedthreshold level of similarity. Comparatively, smaller computed distancevalues that are less than the threshold level denote a smaller distanceor smaller assessed difference between B1 and T1 and may becharacterized as having a higher degree or level of similarity than thatdenoted by the threshold (e.g., such smaller computed distance valuesexceed the minimum required similarity denoted by the threshold). Largercomputed distance values that are more than the threshold level denote alarger distance or larger assessed difference between B1 and T1 and maybe characterized as having a lower degree or level of similarity thanthat denoted by the threshold (e.g., such larger computed distancevalues do not meet the minimum required similarity denoted by thethreshold). In at least one embodiment, if D(x,y) (e.g., as computedusing EQUATION 1), is less than the threshold, then blocks x and y aresimilar; and otherwise x and y are determined as not similar. If twodata blocks are evaluated as being similar using the threshold and adistance computed using EQUATION 1 with respect to the two data blocks,it is expected that at least one sub-block of the two data blocks willmatch when the contents of the two data blocks are examined (e.g., suchas by data content comparison at the sub-block level).

In at least one embodiment, the distance function such as expressed inEQUATION 1 may be applied to the candidate block B1 and the target blockT1 as noted above. To further illustrate, reference is made to the FIG.12.

The example 1200 of FIG. 12 includes the candidate block B1 1210, targetblock T1 1220 and XOR results per sub-block 1230. The element 1202denotes the 8 sub-block or sector positions of the data blocks B1 andT1. T1 may be an existing target block and B1 may be a candidate blockfor which deduplication processing is being performed. In at least oneembodiment, full block deduplication processing for B1 has beenperformed and failed to find a matching target block (e.g., B1 is notfully dedupable). As such, processing may now be performed using thetechniques herein in connection with partial or sub-block deduplicationprocessing. The distance function as in EQUATION 1 may be used todetermine whether to perform partial deduplication processing of B1 withrespect to T1. Using the distance function of EQUATION 1, processing maycompute a distance of B1 with respect to T1, denoting a measurement ofsimilarity between B1 and T1. If the distance as computed using EQUATION1 is less than a specified threshold, B1 and T1 are similar and thecontents of B1 is expected to be similar to the contents of T1. Incontrast, if the distance as computed using EQUATION 1 is not less thanthe specified threshold, B1 and T1 are not similar and the contents ofB1 is not expected to be similar to the contents of T1. Thus, thedistance computed using EQUATION 1 is a metric that provides ameasurement denoting the degree of similarity between B1 and T1. Thecomputed distance may be used to decide whether to further examine B1and T1 to determine any sub-block of T1 matching a sub-block of B1.

With reference to FIG. 12, the element 1230 denotes the XOR operationresults of B1 1210 and T1 1220 per sub-block for each of the 8 sub-blockpositions, where a value of 0 in the i^(th) cell or entry of 1230denotes that contents of the i^(th) sub-block of B1 and T1 are the same,and a value of 1 in the i^(th) cell or entry of 1230 denotes thatcontents of the i^(th) sub-block of B1 and T1 are different. The element1240 denotes that a single entry of 1230 for sub-block 2 has a valueof 1. Based on EQUATION 1, the computed distance D is 1/8=0.125. Assumethe specified threshold is, for example, 0.50. Thus, the computeddistance D=0.125 is less than the specified threshold of 0.50, wherebyB1 and T1 are determined as similar and have at least the specifieddegree of similarity indicated by the threshold.

Responsive to determining the computed distance D(B1, T1) is less thanthe threshold of 0.50, processing may determine that B1 and T1 aresimilar whereby partial deduplication processing may examine sub-blocksof T1 to determine whether one or more sub-blocks of B1 match one ormore sub-blocks of T1. In at least one embodiment, partial or sub-blockdeduplication may be performed with respect to B1 and T1 as describedabove such as in connection with FIG. 9. For example, processing mayexamine the computed XOR results of 1230 to look for those sub-blocks orsectors having a resulting XOR value of 0 denoting that such sub-blocksof B1 and T1 match whereby such sub-blocks of B1 may be deduplicatedsince they are duplicates of T1. In the example 1200, processing maydetermine that all sub-blocks of B1 may be deduplicated as duplicates ofsub-blocks of T1 except for sub-block 2 (having a 1 in the correspondingsub-block position 2 of the XOR results 1230). Processing may beperformed, for example, as described in connection with steps 960 and970 of FIG. 9 and may identify two duplicate ranges of B1—range 1 forsub-blocks 0-1 of B1 that match sub-blocks 0-1 of T1; and range 2 forsub-bocks 3-7 of B1 that match sub-blocks 3-7 of T1. Additionally, suchprocessing may identify a unique region or range for sub-block 2 of B1that cannot be deduplicated.

If no match is found between a sub-block digest for candidate B1 and anL or R digest for T1, partial deduplication processing may proceed toexamine another existing target block T2 and assess whether to performpartial deduplication processing of B1 with respect to T2. Thus, T2 maybe processed and assessed in a manner similar to that as describedherein for the target block T1 as in connection with FIG. 12.

In at least one embodiment, calculation of the distance based onEQUATION 1 may be performed prior to perform full block deduplicationprocessing. If the computed distance D(B1, T1)=0, it means that B1=T1whereby B1 is a full block duplicate of T1, and B1 may be completelydeduplicated at the data block level as a duplicate of T1. In this case,no partial or sub-block deduplication is needed.

It should be noted that computation of the distance with respect to the2 blocks T1 and B1 based on EQUATION 1 requires use of the actual datacontent of both blocks. It may be, for example, that although computeddigests for T1 are in a cached copy of the dedupe DB 150, the actualdata content of T1 is not and may therefore need to be retrieved frompersistent storage, such as non-volatile backend storage 190 (of FIG.1).

As such what will now be described is another embodiment in accordancewith the techniques herein that may be characterized as a variation ofthe distance function as described above in connection with the EQUATION1 and the FIG. 12. Rather than use the actual data content of the datablocks in connection with computing the distance D for two data blocks,x and y, what will now be described is another embodiment in which thedistance denoting similarities between data blocks x and y is determinedusing full block digests for x and y, where the full block digests arecomputed using a distance preserving hash function (described in moredetail below). In such an embodiment, the distance may be computed usingthe full block digests for x and y thereby not requiring access to theactual data content of both x and y at the time the distance iscomputed. Such a computed distance with respect to blocks x and y may beused to draw an inference about the similarities within the two blocks(e.g., regarding the data content of the 2 blocks) without comparing thecontents of the two blocks.

In such an embodiment, a hash function or algorithm may be used tocompute another type of full block digest entry referred to herein as aW type digest entry. A W type digest entry may include a digest computedby a hash function or algorithm using all the content for the data block(e.g., all 8 sub-blocks) where the hash function or algorithm may becharacterized as a distance preserving hash function described in moredetail below. The W type entry may be a digest entry included in thededupe DB 150 including fields as described herein, for example, for Ftype digest entries with the difference that a W type entry includes adigest computed using a distance preserving hash function H. A W typeentry may be computed for each target block and stored in the dedupe DB150. More generally, the distance preserving hash function may be usedto compute hash values or digests for data blocks where such hash valuesor digests are used in connection with computing distances (e.g., basedon EQUATION 2 described below).

In at least one embodiment, different hash functions or algorithms maybe used in computing full block hashes or digests, such as digests of Ftype entries in the dedupe DB 150 used in connection with full blockdeduplication and other full block hashes, and such as digests of W typeentries in the dedupe DB 150 used in connection with computing hashes ordigests used with distance calculations (e.g., such as in the EQUATION 2and FIG. 13 described below). In connection with computing digests orhashes of blocks used with full block deduplication, a first hashfunction may be used that is not a distance preserving hash function (asdescribed in more detail below). Generally, such a first hash functionmay be a stronger hash function in comparison to a second hash functionthat is distance preserving and used to compute hashes or digests usedwith distance computation. The first hash function may be selected witha goal of minimizing hash collisions and may therefore have a lowprobability of hash collision (e.g., such as in comparison to the seconddistance preserving hash function). For example, the first hash functionmay be a cryptographic hash function, such as SHA-1 or SHA-256 that isresistant to hash collisions. In contrast, the second hash function thatis a distance preserving hash function may be a non-cryptographic hashfunction that is less computationally expensive and weaker relative tothe first hash function and therefore the second hash function may havea higher hash collision probability that the first hash function. Forexample, in at least one embodiment, a hash function may compute a CRC(cyclic redundancy check) value for a data block where such a hashfunction may be used as the distance preserving hash function and wherethe generated CRC value for a data block may be used as the hash valueor digest for the data block in distance calculations.

To measure the distance D between two data blocks, a hash function H,also referred to herein as a distance preserving hash function, may beused that preserves the distance property between data blocks such thatthe distance between the hashes of the two blocks will correspond to thedistance (e.g., similarity or differences) between the bits of the twoblocks. The distance function between the data blocks and the hashes ofsuch data blocks preserve the relation between two pairs of data blockswith similar content to each other. For example, assume two blocks withbit contents b 1 and b2 that are to be compared for similarities bymeasuring the distance, (b1,b2) between them. The hash function H may beused to compute the full block hashes of type W that preserves thedistance relation between D(b1,b2) and D(H(b1),H(b2)). The distancefunction between two blocks and the distance function used for the twohashes of such blocks may be different and still preserve the relationbetween hashes and data chunks.

Lemma 1 may be defined as follows:

If (b 1, b2 and b3) are three blocks of the same size; and

D₁₂ and D₁₃ are the corresponding distances of each pair, where D₁₂=D(b1,b2) is the distance between b1 and b2, and D₁₃=D(b1, b3) is thedistance between b1 and b3, then a distance preserving hash function Hmay be defined to determine hash values for all 3 blocks, b1, b2, and b3where:

-   -   H1=H(b1) is the computed hash for b1 using the hash function H;    -   H2=H(b2) is the computed hash for b2 using the hash function H;    -   H3=H(b3) is the computed hash for b3 using the hash function H;    -   d₁₂=D(H1, H2) and d₁₃=D(H1, H3) (e.g., distances between hash        values); and    -   if d₁₂<d₁₃ then D₁₂<D₁₃        Thus, based on Lemma 1 above, a distance preserving hash        function has the property:    -   if d₁₂<d₁₃ then D₁₂<D₁₃        whereby the hash function H preserves the distance relation        between the data blocks (e.g., D(b1, b2)) and the hash values of        such data blocks (e.g., D(H(b1), H(b2))). In other words, the        relative distance between blocks are preserved when computing        the distance using the calculated hash values for such blocks.

In at least one embodiment, another distance function DD may be definedusing hash values or digests computed using a distance preserving hashfunction H as follows:DD(Hx,Hy)=(Σ_(i=0) ^(N-1) Hx(i)XOR Hy(i))/N  EQUATION 2Where

H is a distance preserving hash function;

Hx is the computed hash value using hash function H for the data blockx;

Hy is the computed hash value using hash function H for the data blocky;

N is the number of bytes in each of the computed hash values Hx and Hy,assuming 0-based byte numbering;

Hx(i) denotes byte “i” in the computed hash value Hx for the data blockx;

Hy(i) denotes byte “i” in the computed hash value Hy for the data blocky; and

XOR denotes the logical exclusive OR operation as applied tocorresponding bit positions of the two corresponding bytes of thecomputed hash values, Hx(i) for data block x and Hy(i) for data block y.

Generally, based on EQUATION 2, the XOR function is applied to the hashvalues of the data blocks x and y rather than, as in EQUATION 1, wherethe XOR function is applied to the data blocks themselves.

It should be noted that the numerator and the denominator of EQUATION 2are with respect to the total number of bytes in the computed hashvalues Hx, Hy, where the numerator denotes how many of the bytes of suchcomputed hash values Hx, Hy are different. As a variation, an embodimentmay compute the distance with respect to the total number of bitpositions or other grouping of bits in the compute hash values Hx, Hy.

The distance computed using EQUATION 2 may be characterized as anormalized value based on the total number of bytes in each hash value.It will be appreciated by those skilled in the art that furthervariations of techniques herein may use computed distances that are notnormalized. For example, an embodiment in accordance with the techniquesherein may simply use the numerator computed based on EQUATION 2 as thecomputed distance values where such computed distances are notnormalized. In such an embodiment, the specified threshold discussedelsewhere herein may be an integer threshold quantity.

In at least one embodiment in accordance with the techniques herein, adecision may be made using the distance function DD(Hx,Hy) as to whetherto look for partial or sub-block matches for the candidate block B1within that particular target block T1, where x=the candidate block B1,Hx is the hash value for B1 computed using the distance preserving hashfunction H, y=the target block T1, and Hy is the hash value for T1computed using the distance preserving hash function H. In this manner,using the distance preserving hash function H and computed hash values,the distance function DD of EQUATION 2 denotes a measurement ofsimilarity between the blocks B1 and T1. For example, lower distancevalues may denote a higher degree of similarity between B1 and T1. If DD(H(B1), H(T1)) of EQUATION 2 is less than a specified threshold distancedenoting a threshold level of similarity, then it may be determined thatB1 and T1 have at least the specified threshold level of similarity.Comparatively, smaller computed distance values that are less than thethreshold level denote a smaller distance or smaller assessed differencebetween B1 and T1 and may be characterized as having a higher degree orlevel of similarity than that denoted by the threshold (e.g., suchsmaller computed distance values exceed the minimum required similaritydenoted by the threshold). Larger computed distance values that are morethan the threshold level denote a larger distance or larger assesseddifference between B1 and T1 and may be characterized as having a lowerdegree or level of similarity than that denoted by the threshold (e.g.,such larger computed distance values do not meet the minimum requiredsimilarity denoted by the threshold). In at least one embodiment, ifDD(H(B1), H(T1)) (e.g., as computed using EQUATION 2), is less than thethreshold, then blocks B1 and T1 are similar; and otherwise B1 and T1are determined as not similar.

In at least one embodiment, the distance function such as expressed inEQUATION 2 may be applied to the candidate block B1 and the target blockT1 as noted above. To further illustrate, reference is made to the FIG.13. In the example 1300 of FIG. 13, a hash value that is 20 bytes inlength is used for purposes of illustration. More generally, anembodiment may use any suitable size hash value as the digests for theblocks.

The example 1300 of FIG. 13 includes the hash value H1=H(candidate blockB1) 1310, H2=H(target block T1) 1320, and the XOR results per byte 1330.The element 1302 denotes the 20 bytes of the computed hash values H1,H2, respectively, for the data blocks B1 and T1. As described inconnection with FIG. 12, T1 may be an existing target block and B1 maybe a candidate block for which deduplication processing is beingperformed. In at least one embodiment, full block deduplicationprocessing for B1 has been performed and failed to find a matchingtarget block (e.g., B1 is not fully dedupable). As such, processing maynow be performed using the techniques herein in connection with partialor sub-block deduplication processing. The distance function as inEQUATION 2 may be used to determine whether to perform partialdeduplication processing of B1 with respect to T1. Using the distancefunction of EQUATION 2, processing may compute a distance with respectto the hash values of B1 and T1, denoting a measurement of similaritybetween B1 and T1. If the distance as computed using EQUATION 2 is lessthan a specified threshold, B1 and T1 are similar and the contents of B1is expected to be similar to the contents of T1. In contrast, if thedistance as computed using EQUATION 2 is not less than the specifiedthreshold, B1 and T1 are not similar and the contents of B1 is notexpected to be similar to the contents of T1. Thus, the distancecomputed using EQUATION 2 is a metric that provides a measurementdenoting the degree of similarity between B1 and T1.

With reference to FIG. 13 the element 1330 denotes the XOR operationresults of H1 1310 and H2 1320 per byte for each of the 20 bytepositions, where a value of 0 in the i^(th) cell or entry of 1330denotes that contents of the i^(th) byte of H1 and H2 are the same, anda value of 1 in the i^(th) cell or entry of 1330 denotes that contentsof the i^(th) byte of H1 and H2 are different. The element 1340 denotesthat 4 entries of 1230 for sub-blocks 2, 8, 12 and 18 each have a valueof 1. Based on EQUATION 2, the computed distance DD is 4/20=0.20. Assumethe specified threshold is, for example, 0.50. Thus, the computeddistance DD=0.20 is less than the specified threshold of 0.50, wherebyB1 and T1 are determined as similar and have at least the specifieddegree of similarity indicated by the threshold.

Responsive to determining the computed distance DD(H1, H2) is less thanthe threshold of 0.50, processing may determine that B1 and T1 aresimilar whereby partial deduplication processing may examine sub-blocksof T1 to determine whether one or more sub-blocks of B1 match one ormore sub-blocks of T1. In at least one embodiment, partial or sub-blockdeduplication may be performed with respect to B1 and T1 in a mannersimilar to that as described above such as in connection with FIG. 9.For example, processing may compute sub-block digests for the sub-blocksof the candidate B1 (e.g., step 930) and compare (e.g., step 940) suchcomputed sub-block digests for the candidate B1 to the L and R digestsfor the L and R sub-blocks of T1 as stored in the dedupe DB 150. If amatch is found between a sub-block digest for candidate B1 and an L or Rdigest for T1, (step 950 evaluates to yes), processing of step 960 and970 of FIG. 9 may be performed. If no match is found between a sub-blockdigest for candidate B1 and an L or R digest for T1, partialdeduplication processing may proceed to examine another existing targetblock T2 and assess whether to perform partial deduplication processingof B1 with respect to T2. Thus, T2 may be processed and assessed in amanner similar that as described herein for the target block T1 as inconnection with FIG. 13.

As a variation to that as described above, responsive to determining thedistance between B1 and T1 as computed using EQUATION 2 is less than thethreshold, an embodiment may perform partial or sub-block deduplicationof B1 with respect to T1 by simply performing a byte-by-byte comparisonof B1 and T1 to determine which bytes of B1 and T1 match. In this lattervariation, an embodiment may omit calculating the sub-block leveldigests for the sub-blocks of B1 and may omit comparing such sub-blockdigests for B1 to sub-block L and R digests for T1 as stored in thededupe DB 150. Processing may be performed, for example, as described inconnection with steps 960 and 970 of FIG. 9 to identify one or moreduplicate sub-block ranges of B1 that match sub-blocks of T1.Additionally, such processing may identify one or more unique regions orranges of B1 that cannot be deduplicated, such as using sub-blocks ofT1.

In at least one embodiment, full block deduplication processing may beperformed prior to calculation of the distance based on EQUATION 2 andprior to perform partial or sub-block level deduplication processing.For example, full block deduplication processing may be performed bycalculating a first full block digest for B1, and then comparing thefirst full block digest for B1 to other full block digests of F typeentries for target blocks stored in the dedupe DB 150. Such full blockdeduplication processing may be unsuccessful in full block deduplicationof B1 (e.g., failed to locate a digest of an F type entry for a targetblock in the dedupe DB 150 matching the first full block digest for B1).If the candidate block B1 does not match an existing target block,partial or sub-block deduplication may be performed, such as using thecomputed distance DD of EQUATION 2 with respect to the candidate blockB1 and one or more target blocks. If the computed distance DD ofEQUATION 1 with respect to a particular target block T is less than aspecified threshold, then further processing may be performed to locatesub-blocks of B1 matching sub-blocks of T whereby B1 may be partiallydeduplicated using one or more sub-blocks of T1.

Generally, any suitable hash function or algorithm that is a distancepreserving hash function as described herein may be used in anembodiment in accordance with the techniques herein. For example, asnoted above in at least one embodiment, the distance preserving hashfunction used in connection with computing digests or hashes used withdistances (e.g., as in EQUATION 2) may be a CRC function that computes aCRC value or checksum value for a data block.

Computation of CRC values or checksum values for input data, such asdata blocks, using a CRC function is known in the art. CRC is a checksumalgorithm typically used to detect inconsistencies of data during datatransmission. A checksum, calculated by CRC, is typically attached tothe data during data transmission. CRC is generally based on divisionwhere the input data block is interpreted as a binary bit stream (e.g.,dividend) that is divided by another fixed binary number (e.g.,divisor). The remainder of this division is the checksum value. Thebinary numbers used as the dividend and divisor are treated as binarypolynomials where the bits of such binary numbers are used ascoefficients of the polynomials. The divisor may also be referred to asa generator polynomial that is statically defined using the CRCalgorithm, where CRC-n denotes use of a fixed defined generatorpolynomial with “n+1” bits. Generally, the CRC checksum may be definedas the remainder after repeatedly performing XOR operations using thegenerator polynomial or divisor.

A CRC function is typically used in computing CRC values or checksumvalues used as error correcting codes. The CRC function is lessfrequently, and not so typically, used as a hashing algorithm due to thefact that the collision probability is higher with a CRC hash functionthan other hashing algorithms, such as cryptographic hashing. In otherwords, in comparison to stronger hashing algorithms, the probability isgreater using a CRC hash function that two different data blocks willhave the same CRC value. For example, the first bit sequence 1011011001110000 and the second bit sequence 10010110 01110000 differ by one bityet the CRC of both (e.g., CRC=01110000) are identical due to thecorrection factor for the changed bit. However, from the perspective ofidentifying similarities between blocks in at least one embodiment, itis desirable to use CRC values as the hashes or digests for distancecalculations, such as described herein with EQUATION 2. The CRC cancorrect only a certain number of bits meaning that similar data blockshave corresponding CRC values where a majority of the bits may beidentical except for a small number of different bits. Thus, unlikecryptographic hash functions that may be used for full blockdeduplication in at least one embodiment, CRC is an easily reversiblefunction which makes it suitable for detecting similarities between datablocks. However, in at least one embodiment, a hash function thatcalculates CRC values may not be suitable for digests of blocks used inconnection with full block deduplication.

What will now be described are flowcharts of processing steps that maybe performed in an embodiment in accordance with the techniques hereinwhere such flowcharts summarize processing described above.

Referring to FIG. 14, shown is a first flowchart 1400 of processingsteps that may be performed in an embodiment in accordance with thetechniques herein. The flowchart 1400 indicates processing steps thatmay be performed for a single candidate block whereby the steps of theflowchart 1400 may be repeated for subsequent candidate blocks.

At the step 1402, a candidate block is received. From the step 1402,control proceeds to the step 1404. At the step 1404, full blockdeduplication processing is performed for the candidate where thecandidate is determined as a full block match of an existing targetblock. From the step 1404, processing proceeds to the step 1406. At thestep 1406, a determination is made as to whether full blockdeduplication processing of step 1404 successfully located an existingtarget block that matches the candidate block. If the step 1406evaluates to yes, control proceeds to the step 1408 where the candidateblock is stored as a full deduplicated block of the matching targetblock.

If the step 1406 evaluates to no, control proceeds to the step 1410. Atthe step 1410, partial or sub-block deduplication processing isperformed for the candidate block. From the step 1410, control proceedsto the step 1412. At the step 1412, a determination is made as towhether partial or sub-block deduplication processing of step 1410successfully located an existing target block that having at least onesub-block matching at least one sub-block of the candidate block. If thestep 1412 evaluates to yes, control proceeds to the step 1414 to storethe candidate block as a partially deduplicated block having at leastone sub-block that matches a sub-block of an existing target block. Itshould be noted that an embodiment may allow a single candidate block tobe partially deduplicated with respect to one or more existing targetblocks. If the step 1412 evaluates to no, control proceeds to the step1416 to store the candidate as a new target block. The step 1416 mayinclude persistently storing content of the candidate block on physicalstorage and also in cache. The step 1416 may include calculating one ormore digest values for the new target block and storing such digestvalues as new entries in the dedupe DB 150. The one or more digestvalues may include an F type digest entry, and a W type digest entry forthe new target block. Depending on the embodiment and the particularpartial deduplication processing performed, the step 1416 may includestoring an R type digest entry and an L type digest entry for the newtarget block.

Referring to FIG. 15, shown is a second flowchart 1500 of processingsteps that may be performed in an embodiment in accordance with thetechniques herein. The flowchart 1500 provides additional detailregarding one particular embodiment in which the distance function iscomputed, such as in EQUATION 1, based on the XOR of the data blocks B1and T1. The flowchart 1500 indicates processing steps that may beperformed for a single candidate block whereby the steps of theflowchart 1500 may be repeated for subsequent candidate blocks.

At the step 1502, the candidate block B1 and a target block T1 areobtained. From the step 1502, control proceeds to the step 1504 whereprocessing is performed to compute the bit-wise XOR result of B1 XORedwith T1. From the step 1504, control proceeds to the step 1506. At thestep 1506, the distance between B1 and T1 is calculated as in EQUATION 1based on the computed XOR results from step 1504. From the step 1506,control proceeds to the step 1508. At the step 1508, a determination ismade as to whether the distance computed in step 1506 is zero. If thestep 1508 evaluates to yes, control proceeds to the step 1510. At thestep 1510, it is determined that B1 and T1 match whereby B1 is fullydeduplicated as a full block match for T1. The step 1510 is similar tothe step 1408 of FIG. 14.

If the step 1508 evaluates to no, control proceeds to the step 1512where a determination is made as to whether the distance is less than aspecified threshold. Consistent with discussion herein, the distancedenotes a measurement of similarity between the candidate data block B1and a target data block T1. If the step 1512 evaluates to no, controlproceeds to the step 1514 where it is determined that B1 and T1 are notsimilar. In connection with the step 1514, processing does not furtherexamine T1 for a possible matching sub-block in B1. From the step 1514,control proceeds to the step 1522 where a determination is made as towhether processing of all existing targets has completed. If step 1522evaluates to yes, control proceeds to the step 1526 where the candidateis not partially deduplicated and where the candidate block is stored asa new target block as denoted in the step 1416 of the FIG. 14. If thestep 1522 evaluates to no, control proceeds to the step 1524 to obtainthe next target block. From the step 1524, control proceeds to the step1504 to continue partial or sub-block deduplication processing using thenext target block as the current target block T1.

If the step 1512 evaluates to yes, control proceeds to the step 1516where it is determined that B1 and T1 are similar. The step 1516 mayinclude performing partial or sub-block deduplication for B1 withrespect to T1. The step 1516 may include determining what sub-blocks ofB1 can be stored as duplicates of the existing target block T1. From thestep 1516, control proceeds to the step 1518 where a determination ismade as to whether partial deduplication of B1 with respect to T1 wassuccessful. Put another way, at the step 1518, a determination is madeas to whether at least one sub-block of B1 can be deduplicated wherebythe at least one sub-block of B1 is stored as a matching sub-block ofT1. If the step 1518 evaluates to yes, control proceeds to the step1520. At the step 1520, processing is performed to store the candidateblock B1 as a partially deduplicated block having at least one sub-blockmatching at least one sub-block of the target block T1. The step 1520 issimilar to the step 1414 of FIG. 14. If the step 1518 evaluates to no,control proceeds to the step 1522.

Referring to FIG. 16, shown is third flowchart 1600 of processing stepsthat may be performed in an embodiment in accordance with the techniquesherein. The flowchart 1600 provides additional detail regarding oneparticular embodiment in which the distance function is computed, suchas in EQUATION 2, based on the XOR of the hash values for the datablocks B1 and T1, where such hash values are computed using a distancepreserving hash function. The flowchart 1600 indicates processing stepsthat may be performed for a single candidate block whereby the steps ofthe flowchart 1600 may be repeated for subsequent candidate blocks.

At the step 1602, the candidate block B1 is obtained. The step 1602includes computing an F type digest for the candidate block B1. Asdescribed herein the F type digest for B1 may be computed using a hashfunction and the full block content of B1. The hash function used incomputing the F type digest for B1 is a first hash function that isdifferent than the second hash function H using to compute hash valuesused in determining the distance, such as in EQUATION 2. The first hashfunction may be stronger than the distance preserving hash function Hused as the second hash function.

From the step 1602, control proceeds to the step 1603 where processingis performed to check each F type entry of the dedupe DB 150 for a matchto B1. Step 1603 may include determining if there is an existing F typeentry in the dedupe DB having a digest that matches the computed fullblock F type digest for B1 (as computed in the step 1602), and if so,comparing the actual content of the target data block having thematching entry to the content of B1. From the step 1603, controlproceeds to the step 1604. At the step 1604, a determination is made asto whether a matching target block in the dedupe DB was found for thecandidate block B1. If the step 1604 evaluates to yes, control proceedsto the step 1610. At the step 1610, B1 is processed as a fulldeduplicated block of the matching target block. The step 1610 issimilar to the step 1408 of the FIG. 14. If the step 1604 evaluates tono, control proceeds to the step 1605 where processing is performed tocompute H(B1) using the distance preserving hash function H. From thestep 1605, control proceeds to the step 1606. At the step 1606, a targetblock T1 is obtained and H(T1) is computed. From the step 1606, controlproceeds to the step 1607.

At the step 1607, processing is performed to calculate the distancebetween H(B1) and H(T1), such as using EQUATION 2. From the step 1607,control proceeds to the step 1612 where a determination is made as towhether the distance is less than a specified threshold. Consistent withdiscussion herein, the distance denotes a measurement of similaritybetween the candidate data block B1 and the target data block T1. If thestep 1612 evaluates to no, control proceeds to the step 1614 where it isdetermined that B1 and T1 are not similar. In connection with the step1614, processing does not further examine T1 for a possible matchingsub-block in B1. From the step 1614, control proceeds to the step 1622where a determination is made as to whether processing of all existingtargets has completed. If step 1622 evaluates to yes, control proceedsto the step 1626 where the candidate is not partially deduplicated andwhere the candidate block is stored as a new target block as denoted inthe step 1416 of the FIG. 14. If the step 1622 evaluates to no, controlproceeds to the step 1624 to obtain the next target block. From the step1624, control proceeds to the step 1504 to continue partial or sub-blockdeduplication processing using the next target block as the currenttarget block T1.

If the step 1612 evaluates to yes, control proceeds to the step 1616where it is determined that B1 and T1 are similar. The step 1616 mayinclude performing partial or sub-block deduplication for B1 withrespect to T1. The step 1616 may include determining what sub-blocks ofB1 can be stored as duplicates of the existing target block T1. From thestep 1616, control proceeds to the step 1618 where a determination ismade as to whether partial deduplication of B1 with respect to T1 wassuccessful. Put another way, at the step 1618, a determination is madeas to whether at least one sub-block of B1 can be deduplicated wherebythe at least one sub-block of B1 can be stored as a matching sub-blockof T1. If the step 1618 evaluates to yes, control proceeds to the step1620. At the step 1620, processing is performed to store the candidateblock B1 as a partially deduplicated block having at least one sub-blockmatching at least one sub-block of the target block T1. The step 1620 issimilar to the step 1414 of FIG. 14. If the step 1618 evaluates to no,control proceeds to the step 1622.

The techniques herein may be performed by any suitable hardware and/orsoftware. For example, the techniques herein may be performed byexecuting code which is stored on any one or more different forms ofcomputer-readable media, where the code may be executed by one or moreprocessors, for example, such as processors of a computer or othersystem, an ASIC (application specific integrated circuit), and the like.Computer-readable media may include different forms of volatile (e.g.,RAM) and non-volatile (e.g., ROM, flash memory, magnetic or opticaldisks, or tape) storage which may be removable or non-removable.

While the invention has been disclosed in connection with embodimentsshown and described in detail, their modifications and improvementsthereon will become readily apparent to those skilled in the art.Accordingly, the spirit and scope of the present invention should belimited only by the following claims.

What is claimed is:
 1. A method of processing data comprising:receiving, using a processor, a candidate data block; computing, using aprocessor, a distance using a distance function, wherein the distancedenotes a measurement of similarity between the candidate data block anda target data block, wherein the distance computed by the distancefunction is less than a second distance computed by the distancefunction for another pair of data blocks, and the distance denotes thatthe candidate data block and the target data block have a higher degreeof similarity than the another pair of data blocks having the seconddistance; determining, using a processor, that the candidate data blockis not a duplicate of the target block; and responsive to determiningthe candidate data block is not a duplicate of the target block,determining using a processor whether the distance is less than athreshold distance denoting a threshold level of similarity, andresponsive to determining that the distance is less than the thresholddistance, performing first processing using a processor including:determining that the candidate data block and the target block have alevel of similarity greater than the threshold level of similaritydenoted by the threshold distance; and performing partial datadeduplication of the candidate data block with respect to the targetdata block to identify at least one sub-block of the candidate datablock that is a duplicate of at least one sub-block of the target datablock, wherein said partial data deduplication includes: determiningwhether at least one sub-block of the candidate data block matches atleast one sub-block of the target data block; and responsive todetermining at least one sub-block of the candidate data block matchesat least one sub-block of the target data block, performing secondprocessing that stores the candidate data block as a partiallydeduplicated data block with at least one sub-block that is a duplicateof another sub-block of the target data block.
 2. The method of claim 1,wherein the distance function computes the distance based on results ofa logical exclusive-or operation of the candidate data block and thetarget data block.
 3. The method of claim 2, wherein the distancefunction computes a normalized distance value as the distance.
 4. Themethod of claim 2, further comprising: responsive to determining thedistance is less than the threshold distance, determining the candidatedata block is similar to the target data block.
 5. The method of claim2, further comprising: determining whether the distance is zero; andresponsive to determining the distance is zero, determining that thecandidate data block is a duplicate of the target data block and storingthe candidate data block as a duplicate of the target data block.
 6. Themethod of claim 1, wherein the distance function computes the distancebased on results of a bit-wise logical exclusive-or operation of a firstdigest and a second digest, wherein the first digest is computed for thecandidate data block using a distance preserving hash function andwherein the second digest is computed for the target data block usingthe distance preserving hash function.
 7. The method of claim 6, whereinthe distance function computes a normalized distance value as thedistance.
 8. The method of claim 6, further comprising: responsive todetermining the distance is less than the threshold distance,determining the candidate data block is similar to the target datablock.
 9. The method of claim 6, further comprising: determining whethera third digest matches a fourth digest, wherein the third digest iscomputed for the candidate data block using a second hash function andwherein the fourth digest is computed for the target data block usingthe second hash function, wherein the second hash function is a strongerhash function than the first hash function; responsive to determiningthe third digest matches the fourth digest, comparing the candidate datablock to the target data block; determining, in accordance with saidcomparing whether the candidate data block matches the target datablock; and responsive to determining the candidate data block matchesthe target data block, storing the candidate data block as a fullydeduplicated data block.
 10. The method of claim 9, wherein the firsthash function is a non-cryptographic hash function and the second hashfunction is a cryptographic hash function.
 11. The method of claim 10,wherein the first hash function, that is the distance preserving hashfunction, is a cyclic redundancy check (CRC) function that computes afirst checksum as the first digest for the candidate data block andcomputes a second checksum as the second digest for the target datablock.
 12. The method of claim 1, wherein the method is performed aspart of inline processing of the candidate data block in connection withan I/O path or data path when servicing an I/O accessing the candidatedata block.
 13. The method of claim 1, wherein the method is performedoffline and not as part of inline processing of the candidate data blockin connection with an I/O path or data path when servicing an I/Oaccessing the candidate data block.
 14. A system comprising: aprocessor; and a memory comprising code stored thereon that, whenexecuted, performs a method of processing data comprising: receiving acandidate data block; computing a distance using a distance function,wherein the distance denotes a measurement of similarity between thecandidate data block and a target data block, wherein the distancecomputed by the distance function is less than a second distancecomputed by the distance function for another pair of data blocks, andthe distance denotes that the candidate data block and the target datablock have a higher degree of similarity than the another pair of datablocks having the second distance; determining that the candidate datablock is not a duplicate of the target block; and responsive todetermining the candidate data block is not a duplicate of the targetblock, determining whether the distance is less than a thresholddistance denoting a threshold level of similarity, and responsive todetermining that the distance is less than the threshold distance,performing processing including: determining that the candidate datablock and the target block have a level of similarity greater than thethreshold level of similarity denoted by the threshold distance; andperforming partial data deduplication of the candidate data block withrespect to the target data block to identify at least one sub-block ofthe candidate data block that is a duplicate of at least one sub-blockof the target data block, wherein said partial data deduplicationincludes: determining whether at least one sub-block of the candidatedata block matches at least one sub-block of the target data block; andresponsive to determining at least one sub-block of the candidate datablock matches at least one sub-block of the target data block,performing second processing that stores the candidate data block as apartially deduplicated data block with at least one sub-block that is aduplicate of another sub-block of the target data block.
 15. Anon-transitory computer readable medium comprising code stored thereonthat, when executed, performs a method of processing data comprising:receiving a candidate data block; computing a distance using a distancefunction, wherein the distance denotes a measurement of similaritybetween the candidate data block and a target data block, wherein thedistance computed by the distance function is less than a seconddistance computed by the distance function for another pair of datablocks, and the distance denotes that the candidate data block and thetarget data block have a higher degree of similarity than the anotherpair of data blocks having the second distance; determining that thecandidate data block is not a duplicate of the target block; andresponsive to determining the candidate data block is not a duplicate ofthe target block, determining whether the distance is less than athreshold distance denoting a threshold level of similarity, andresponsive to determining that the distance is less than the thresholddistance, performing processing including: determining that thecandidate data block and the target block have a level of similaritygreater than the threshold level of similarity denoted by the thresholddistance; and performing partial data deduplication of the candidatedata block with respect to the target data block to identify at leastone sub-block of the candidate data block that is a duplicate of atleast one sub-block of the target data block, wherein said partial datadeduplication includes: determining whether at least one sub-block ofthe candidate data block matches at least one sub-block of the targetdata block; and responsive to determining at least one sub-block of thecandidate data block matches at least one sub-block of the target datablock, performing second processing that stores the candidate data blockas a partially deduplicated data block with at least one sub-block thatis a duplicate of another sub-block of the target data block.
 16. Amethod of processing data comprising: receiving a candidate data block;computing, using a distance function, a distance between the candidatedata block and a target data block, wherein the distance denotes ameasurement of similarity between the candidate data block and a targetdata block, wherein said computing the distance between the candidatedata block and the target data block includes: computing a first digestfor the candidate data block using a distance preserving hash function;computing a second digest for the target data e block using the distancepreserving hash function; and computing the distance based on a resultof a bit-wise logical exclusive-or operation of the first digest and thesecond digest; determining whether the distance between the candidatedata block and a target data block is zero; responsive to determiningthe distance between the candidate data block and a target data block iszero, determining the candidate data block is a duplicate of the targetdata block and storing the candidate data block as a duplicate of thetarget data block; and responsive to determining the distance betweenthe candidate data block and a target data block is not zero, performingfirst processing including: determining whether the distance is lessthan a threshold distance denoting a threshold level of similarity; andresponsive to determining the distance is less than the thresholddistance, performing second processing including: determining that thecandidate data block and the target block have a level of similaritygreater than the threshold level of similarity denoted by the thresholddistance; and performing partial data deduplication of the candidatedata block with respect to the target data block to identify at leastone sub-block of the candidate data block that is a duplicate of atleast one sub-block of the target data block, wherein said partial datadeduplication includes: determining whether at least one sub-block ofthe candidate data block matches at least one sub-block of the targetdata block; and responsive to determining at least one sub-block of thecandidate data block matches at least one sub-block of the target datablock, performing second processing that stores the candidate data blockas a partially deduplicated data block with at least one sub-block thatis a duplicate of another sub-block of the target data block.
 17. Themethod of claim 16, wherein a first distance value is computed for afirst pair of data blocks using the distance function and a seconddistance value is computed for a second pair of data blocks using thedistance function, and wherein the first distance value is less than thesecond distance value, and the first pair of data blocks are determinedas having a higher level of similarity than the second pair of datablocks.
 18. A system comprising: one or more processors; and one or morememories comprising code stored thereon that, when executed, performs amethod of processing data comprising: receiving a candidate data block;computing, using a distance function, a distance between the candidatedata block and a target data block, wherein the distance denotes ameasurement of similarity between the candidate data block and a targetdata block, wherein said computing the distance between the candidatedata block and the target data block includes: computing a first digestfor the candidate data block using a distance preserving hash function;computing a second digest for the target data e block using the distancepreserving hash function; and computing the distance based on a resultof a bit-wise logical exclusive-or operation of the first digest and thesecond digest; determining whether the distance between the candidatedata block and a target data block is zero; responsive to determiningthe distance between the candidate data block and a target data block iszero, determining the candidate data block is a duplicate of the targetdata block and storing the candidate data block as a duplicate of thetarget data block; and responsive to determining the distance betweenthe candidate data block and a target data block is not zero, performingfirst processing including: determining whether the distance is lessthan a threshold distance denoting a threshold level of similarity; andresponsive to determining the distance is less than the thresholddistance, performing second processing including: determining that thecandidate data block and the target block have a level of similaritygreater than the threshold level of similarity denoted by the thresholddistance; and performing partial data deduplication of the candidatedata block with respect to the target data block to identify at leastone sub-block of the candidate data block that is a duplicate of atleast one sub-block of the target data block, wherein said partial datadeduplication includes:  determining whether at least one sub-block ofthe candidate data block matches at least one sub-block of the targetdata block; and  responsive to determining at least one sub-block of thecandidate data block matches at least one sub-block of the target datablock, performing second processing that stores the candidate data blockas a partially deduplicated data block with at least one sub-block thatis a duplicate of another sub-block of the target data block.
 19. Anon-transitory computer readable medium comprising code stored thereonthat, when executed, performs a method of processing data comprising:receiving a candidate data block; computing, using a distance function,a distance between the candidate data block and a target data block,wherein the distance denotes a measurement of similarity between thecandidate data block and a target data block, wherein said computing thedistance between the candidate data block and the target data blockincludes: computing a first digest for the candidate data block using adistance preserving hash function; computing a second digest for thetarget data e block using the distance preserving hash function; andcomputing the distance based on a result of a bit-wise logicalexclusive-or operation of the first digest and the second digest;determining whether the distance between the candidate data block and atarget data block is zero; responsive to determining the distancebetween the candidate data block and a target data block is zero,determining the candidate data block is a duplicate of the target datablock and storing the candidate data block as a duplicate of the targetdata block; and responsive to determining the distance between thecandidate data block and a target data block is not zero, performingfirst processing including: determining whether the distance is lessthan a threshold distance denoting a threshold level of similarity; andresponsive to determining the distance is less than the thresholddistance, performing second processing including: determining that thecandidate data block and the target block have a level of similaritygreater than the threshold level of similarity denoted by the thresholddistance; and performing partial data deduplication of the candidatedata block with respect to the target data block to identify at leastone sub-block of the candidate data block that is a duplicate of atleast one sub-block of the target data block, wherein said partial datadeduplication includes: determining whether at least one sub-block ofthe candidate data block matches at least one sub-block of the targetdata block; and responsive to determining at least one sub-block of thecandidate data block matches at least one sub-block of the target datablock, performing second processing that stores the candidate data blockas a partially deduplicated data block with at least one sub-block thatis a duplicate of another sub-block of the target data block.